diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 5022176e6..229b5e7f9 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,6 @@ +## 2026-10-23 — d-mpf-math-polyfill-v1 — the mathematical polyfill API, Stage 1 → `.claude/plans/d-mpf-math-polyfill-v1.md` + +**Status:** D-MPF-1 MEASURED, lab only; D-MPF-2.. PROPOSED. The Stage 1 polyfill API is the shipped R2IL procedure layer plus checked scalar arithmetic over mask-risc, with relalg as the rewriting frontend; no new primitive. One expression matches an independent row loop; at 64 K rows it costs under 1 ns/row, at one word ~600 ns per call. Gaps by counterexample: element-wise arithmetic, unsigned compare, floats, one-word fixed cost. Independent of ONNX and of the S0 drafts. Entry `2026-10-23-d-mpf-1-polyfill-probe.md`. ## 2026-10-23 — d-onnx-relalg-cap-capstone-1750-v1 — ONNX intake onto existing lance-graph surfaces (capstone S0, #1750) → `.claude/plans/d-onnx-relalg-cap-capstone-1750-v1.md` **Status:** PROPOSAL, S0 docs only; nothing implemented. The pipeline "ONNX graph → RelAlg polyfill → Cartesian projection → native execution" is false as stated: relalg is a single-table integer selection with Count/Sum/Min/Max/Keep, so it receives only the integer selection/aggregate island; numeric ops route to existing ndarray kernels; Gather/Scatter, control flow and `ai.onnx.ml` are typed refusals. The ONNX `ModelProto` stays the only graph (D-RA-2 = A; no new AST, primitive or executor). The original DR64³ master plan was not found on `main` or 833 branches and is recorded as an OPEN documentation dependency. S1 is blocked on the operator naming one target model and the protobuf fork coordinates. Entry `2026-10-23-d-onnx-relalg-cap-s0-capstone-1750.md`. #1744's S0 (below) kept the base path; the native-first correction (#1753) governs where they differ. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index e9d62fd58..313bfb630 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,15 @@ +## D-MPF — mathematical polyfill API, Stage 1 (2026-10-23) + +Lab only (`crates/math-polyfill-probe`). Plan `.claude/plans/d-mpf-math-polyfill-v1.md`; entry `entries/2026-10-23-d-mpf-1-polyfill-probe.md`. + +| D-id | scope | status | gate / falsifier | +|---|---|---|---| +| **D-MPF-1** | one expression (SUM − k·COUNT, NULL plane) through the existing r2il procedures; oracle; typed refusals; cost split | MEASURED (this branch) | equal to an `i128` row loop at 64..65 537 rows; D1–D3 red; 0.86–0.91 ns/row at 64 K, ~600 ns per call at 64 rows | +| **D-MPF-2** | attribute the one-word fixed cost (~120 ns mask-risc + ~150 ns dialect) | Measured (#1752) | gate not met, reason recorded: floor alone 39 ns at one word; per-row bottleneck is `masked_sum_i32`'s bit walk (plan §7) | +| **D-MPF-3** | element-wise integer arithmetic (G1) | Blocked (owner decision) | `lane_plus_lane_is_refused` turns into an admitted, oracle-checked op | +| **D-MPF-2b** | dense/sparse hybrid masked sum (per-word density threshold) and one mask reused across count/sum/min/max | Measured (#1752) | hybrid th=16 is 2–3.7× (uniform) and 4–5× (clustered) faster than the walk, and never slower; reuse ~2× when sparse; a fused scalar pass is 6× faster at 50 % (plan §9) | +| **D-MPF-4** | R2IL as a router: attribution of the five things named R2IL, a routing table (DIRECT / COMPOSE / LOWER / REFUSE), and a ClassId → LazyLock → prebound-callable probe against direct ndarray | Measured (#1752) | prebound route == direct ndarray at 64..65 536 rows; one resolution per ClassId, 0 route evaluations and 0 allocations in steady state; frame/focus/scope guard 27–28 ns (4 atomic RMWs); three disable runs red (plan §10) | + ## D-ONNX-CAP — ONNX intake onto existing lance-graph surfaces (2026-10-23) PROPOSAL, docs only. Plan `.claude/plans/d-onnx-relalg-cap-capstone-1750-v1.md`; entry `entries/2026-10-23-d-onnx-relalg-cap-s0-capstone-1750.md` (#1750; #1744's S0 below kept the base path). ONNX `ModelProto` stays the only graph; relalg receives only the integer selection/aggregate island; no new primitive, AST or executor. diff --git a/.claude/board/entries/2026-10-23-d-mpf-1-polyfill-probe.md b/.claude/board/entries/2026-10-23-d-mpf-1-polyfill-probe.md new file mode 100644 index 000000000..79808f3b2 --- /dev/null +++ b/.claude/board/entries/2026-10-23-d-mpf-1-polyfill-probe.md @@ -0,0 +1,12 @@ +# 2026-10-23 — E-D-MPF-1: the Stage 1 polyfill already exists; per row it is sub-nanosecond, per one-word call it is ~600 ns + +**Status:** MEASURED, lab only (`crates/math-polyfill-probe`). Plan `.claude/plans/d-mpf-math-polyfill-v1.md`. + +- The mathematical polyfill API is the shipped R2IL procedure layer (`lance-graph-r2il::procs`) plus checked scalar arithmetic in `FoldDialect`, over mask-risc, with relalg as the rewriting frontend. No new primitive, IR or executor. +- One expression, `SUM(v | sel) − k·COUNT(sel)` with a NULL presence plane, written only with those procedures, equals an independent `i128` row loop at 64 / 1 000 / 4 096 / 65 537 rows. 9 tests; disable runs D1–D3 each turn named tests red. +- Typed refusals already exist for lane + lane arithmetic, unsigned compare, overflow and division by zero. +- Cost (ns per call, medians, 3 runs): at 65 536 rows the expression costs 0.86–0.91 ns/row and beats the scalar loop 1.6×; at 64 rows one fold is ~120 ns in mask-risc plus ~150 ns in the dialect, and the scalar loop (~80 ns) beats the whole expression (~600 ns). +- Gaps by counterexample: G1 element-wise value arithmetic, G2 unsigned compare, G3 floats, G4 the one-word fixed cost. +- The ONNX fork `AdaWorldAPI/optimum-onnx-rs` is empty; `AdaWorldAPI/onnx` is not accessible. + +Next gate: D-MPF-2 (attribute the one-word fixed cost), and the owner decision on G1. diff --git a/.claude/board/entries/2026-10-23-d-mpf-2-fixed-cost.md b/.claude/board/entries/2026-10-23-d-mpf-2-fixed-cost.md new file mode 100644 index 000000000..86b097cf7 --- /dev/null +++ b/.claude/board/entries/2026-10-23-d-mpf-2-fixed-cost.md @@ -0,0 +1,11 @@ +# 2026-10-23 — D-MPF-2: where the native fold's cost goes + +**Status:** MEASURED, lab only (`crates/math-polyfill-probe`, PR #1752). Nothing changed outside the probe. + +- Per call at 64 rows: scalar 12 ns; ndarray floor 39; mask-risc +57; `Program::new` +20 and 1 allocation; dialect +50. Total ~172 ns. +- Per row at 64 K rows: mask-risc ≈ ndarray floor. The floor's cost is `ndarray::simd::masked_sum_i32`, a scalar set-bit walk (LLVM: `tzcnt`/`blsr`, no vector register) whose time tracks popcount: 1.7 / 25 / 44 µs at 0 / 50 / 100 % selected. The compare is ~0.09 ns/row. +- Correction to D-MPF-1: "1.6× faster than the scalar loop" compared against the `i128` filter-chain oracle. An equal-work `i64` loop takes ~11 µs at 64 K rows and is ~3× faster than the fold. +- Proposal, not made: density-chosen `masked_sum_i32` body (branch-free above ~16 set bits per word, walk below). The branch-free form equals the oracle and is flat in selectivity; the walk is 7–10× faster on empty words. +- OPEN: whether the mask form beats a scalar loop when the mask is reused or combined (not measured here). + +Plan: `.claude/plans/d-mpf-math-polyfill-v1.md` §7. diff --git a/.claude/board/entries/2026-10-23-d-mpf-4-classid-route.md b/.claude/board/entries/2026-10-23-d-mpf-4-classid-route.md new file mode 100644 index 000000000..2e8059b65 --- /dev/null +++ b/.claude/board/entries/2026-10-23-d-mpf-4-classid-route.md @@ -0,0 +1,18 @@ +# 2026-10-23 — D-MPF-4: R2IL routes once per ClassId; the steady-state cost is the guard, not the route + +**Status:** MEASURED, lab (`crates/math-polyfill-probe/examples/classid_route.rs`). Plan `.claude/plans/d-mpf-math-polyfill-v1.md` §10. No production crate, ABI or execution contract changed. + +- **Attribution:** five different things are called R2IL, and they stay apart: + - r2sleigh's `r2il`: IL types, no executor. + - the machine reading: `r2conc` / `r2sym`, scalar machine state. + - `ogar-r2il`: the opcode table, readings `0xC400` machine / `0xC401` fold. + - `lance-graph-r2il::FoldDialect`: the fold reading only. + - mask-risc / `ndarray::simd` / `simd_runtime`. + + The 94–99 ns and 171–172 ns of D-MPF-2 are the cost of choosing the mask-risc and FoldDialect paths per call. They are not a cost of R2IL routing. +- **Recovered 2–5 ns figures:** each is a direct kernel, not an R2IL call. The one cited as a comparison is `ndarray::simd` AND, AND, popcount over 4 words at 2.2–2.8 ns, against 54–70 ns through r2il + mask-risc (`2026-10-21-d-rft-1-cost-breakdown.md`). +- **Routing table:** every wired fold-reading op has a DIRECT or COMPOSE route onto existing `ndarray::simd` entrypoints, or is refused. Only machine semantics need R2Sleigh: carry/borrow flags, address-space load/store, branches, width-exact integers. +- **Measured** (64 rows / 65 536 rows): prebound route without guard 36–39 / ≈ direct ndarray. Guard (`FrameStore::check` + focus + scope) 27–28 ns, 0 allocations; objdump shows 2× `lock cmpxchg` + 2× `lock decq` from `Arc::downgrade`. Per-call `LazyLock` + `match` adds 0–1 ns. Prepared R2IL (FoldDialect's `last_program` as a `CompiledProgram`) = prepared mask-risc, 85–94 ns. +- **Asserted:** one resolution per ClassId; 0 route evaluations in steady state; LOWER/REFUSE refused at bind; focus, scope and source changes refused at run; rebinding does not re-resolve. Three disable runs red. +- **Proposal (not made, DR lab owner):** compare `Weak::as_ptr` with `Arc::as_ptr` instead of `Weak::ptr_eq(.., &Arc::downgrade(..))`. Same identity semantics, no atomics. Unmeasured. +- OPEN: the program-only half of mask-risc's per-call `precheck`/`validate`; mask-sharing callables for multi-op bindings; the U64 unsigned-compare refusal (`lt_u64_to_mask` exists). diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 3e619d3a2..c8d51b72e 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -455 entries, 2026-08-06 .. 2026-10-23. +458 entries, 2026-08-06 .. 2026-10-23. | date | entry id | finding | file | |---|---|---|---| @@ -39,6 +39,9 @@ exactly one of them, never both. | 2026-10-23 | `D-TGA-5` | | [2026-10-23-d-tga-5-squash-gate-preregistration.md](2026-10-23-d-tga-5-squash-gate-preregistration.md) | | 2026-10-23 | `D-ONNX-RELALG-CAP` | S0 result: ONNX as import membrane, DR64³ plan not found | [2026-10-23-d-onnx-relalg-cap-s0.md](2026-10-23-d-onnx-relalg-cap-s0.md) | | 2026-10-23 | `E-D-ONNX-RELALG-CAP-S0-1` | | [2026-10-23-d-onnx-relalg-cap-s0-capstone-1750.md](2026-10-23-d-onnx-relalg-cap-s0-capstone-1750.md) | +| 2026-10-23 | `D-MPF-4` | | [2026-10-23-d-mpf-4-classid-route.md](2026-10-23-d-mpf-4-classid-route.md) | +| 2026-10-23 | `D-MPF-2` | | [2026-10-23-d-mpf-2-fixed-cost.md](2026-10-23-d-mpf-2-fixed-cost.md) | +| 2026-10-23 | `E-D-MPF-1` | | [2026-10-23-d-mpf-1-polyfill-probe.md](2026-10-23-d-mpf-1-polyfill-probe.md) | | 2026-10-23 | `coresearch-thought-git-markov` | | [2026-10-23-coresearch-thought-git-markov.md](2026-10-23-coresearch-thought-git-markov.md) | | 2026-10-23 | `coresearch-thought-git-markov-cc2` | | [2026-10-23-coresearch-thought-git-markov-cc2.md](2026-10-23-coresearch-thought-git-markov-cc2.md) | | 2026-10-23 | `coresearch-julia-egraphs` | | [2026-10-23-coresearch-julia-egraphs.md](2026-10-23-coresearch-julia-egraphs.md) | diff --git a/.claude/plans/d-mpf-math-polyfill-v1.md b/.claude/plans/d-mpf-math-polyfill-v1.md new file mode 100644 index 000000000..0ba890002 --- /dev/null +++ b/.claude/plans/d-mpf-math-polyfill-v1.md @@ -0,0 +1,277 @@ +# D-MPF — the mathematical polyfill API, Stage 1 (source audit + folding probe) + +**Status:** D-MPF-1 MEASURED, D-MPF-2 MEASURED (§7), D-MPF-2b MEASURED (§9), D-MPF-4 MEASURED (§10); D-MPF-3 (G1) blocked on the owner, lab only (`crates/math-polyfill-probe`). Independent of ONNX and of the open S0 drafts (#1744, #1745, #1750); it does not choose between them. + +> ⊘ **Correction (2026-10-23, §10).** This plan wrote "R2IL" for `lance-graph-r2il::FoldDialect` and its procedures. They are one thing among five that share the name. R2IL is an opcode table with two readings, and FoldDialect is the fold reading only (§10.1). The §7 and §8 figures (94–99 ns compiled mask-risc, 171–172 ns through the procedures) are the cost of choosing those paths per call. They are not a floor for R2IL routing: a route resolved once per ClassId runs at the direct `ndarray::simd` cost (§10.4). + +## 1. Finding: the Stage 1 polyfill API already exists + +The polyfill is the typed procedure layer that already ships over R2IL, not a new API: + +| layer | what it is | where | status | +|---|---|---|---| +| set description + proven rewrites | `Rel` (All/Empty/Plane/Atom/And/Or/Not), `Measure` (Count/Sum/Min/Max/Keep), `normalize`, `symbolic` | `crates/lance-graph-relalg` (algebra in `lance-graph-contract::set_algebra`, D-RA-2 = A) | SHIPPED | +| procedures (named R2IL call sequences, no new opcode) | `select_lt filter_lt select_u32 count exists all differ intersect union keep count_and sum min max and_cmp_i32 group_sum margin is_null is_not_null and_between_i32 in_u32 count_valid avg_parts bool_or bool_and not_exists except count_distinct_sorted count_union_all` | `crates/lance-graph-r2il/src/procs.rs` | SHIPPED | +| checked scalar arithmetic on fold results | `IntAdd IntSub IntMult` (overflow refused), `IntSDiv` (division by zero refused) | `FoldDialect` in `crates/lance-graph-r2il/src/lib.rs` | SHIPPED | +| bulk engine | mask-risc `Program` + terminals `Count Any All MaskedSumI32 MaskedMinI32 MaskedMaxI32 MaskedStridedGroupSum BlendI32 Keep`, ternlog `fuse` | `crates/lance-graph-mask-risc` | SHIPPED | +| contract refusals | `Refusal::{NotTwoValued, NoUniverse, Checked, Ieee, SymFamily, Impure, BasisMismatch, PastUniverse, Bag}` | `crates/lance-graph-contract/src/set_rules.rs` | SHIPPED | +| proof / certified recipe | egglog-proved associativity of Boolean-Distinct composition over DR64, `RecipeCertificate` holds egglog's `ProofStore` + `ProofId` | `crates/dr-math-binding-probe` (#1707, #1740, #1742) | LAB | + +Typing the procedures carry today: domain `i32`/`u32` lanes, `i64` scalars; Boolean set algebra on masks; integer `+` over selected rows; NULL as a presence plane (`WHERE` drops UNKNOWN; full K3 lives in Quack `sql_where`); multiplicity Set (Bag refused by the contract); no float lane; every answer exact. + +**ONNX/protobuf fork:** `AdaWorldAPI/optimum-onnx-rs` exists and is empty (no branch, no commit, checked 2026-10-23). `AdaWorldAPI/onnx` is not accessible to this session. There is no integration surface to audit yet (UNKNOWN until the operator names it). + +## 2. D-MPF-1 — one expression through the existing polyfill + +`crates/math-polyfill-probe` writes one expression using only the procedures and scalar calls above: + + E = SUM(v | sel) − k·COUNT(sel), sel = valid(v) ∧ x ≥ lo ∧ y < hi + +`eval` lowers it onto a lent `FoldDialect` (two folds, then `IntMult`, `IntSub`). `oracle` is a row loop in `i128` that shares no code with the procedures, the dialect or mask-risc. + +**Gates (`tests/polyfill.rs`, 9 tests, all green):** + +- equal to the oracle at 64, 1 000, 4 096 and 65 537 rows, for empty, partial and full selections and multipliers 3, 0, −7, 1; +- anti-vacuity: one case selects nothing, one a strict subset, and NULL rows exist inside the selected range; +- NULL is load-bearing: with every row present the answer changes, and the polyfill still matches; +- relalg's `Query` and the procedures count the same rows (two frontends, one substrate); +- typed refusals, each an existing `FoldError`: lane + lane (`LaneVersusLane`), unsigned compare (`NoUnsignedLaneCompare`), scalar overflow (`Overflow`, with a fitting product accepted), an expression whose exact value leaves `i64` (`Overflow`, the oracle confirms it does not fit), division by zero (`DivisionByZero`). + +**Disable runs (committed first, anchors asserted):** D1 drop the NULL gate → 3 tests red; D2 `≥` → `>` → 2 red; D3 `IntSub` → `IntAdd` → 1 red. + +## 3. Cost split (reported, not gated) + +`cargo run --release --example cost_split`, 4-core container, three runs; ns per call, median of 21 batches. `exec` is mask-risc `execute_into` of the prebuilt count program alone; `count` is selection + `COUNT` through the dialect; `expr` is all of E. + +| rows | bind | exec | count | expr | count+bind | keep+bind | relalg | oracle | expr ns/row | +|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 64 | 21–24 | 116–129 | 271–279 | 576–627 | 307–362 | 299–315 | 2 723–2 934 | 78–86 | 9.0–9.8 | +| 512 | 28–56 | 246–463 | 324–731 | 792–1 867 | 395–904 | 385–952 | 2 937–5 627 | 651–738 | 1.6–3.7 | +| 4 096 | 66–74 | 1 056–1 246 | 982–1 116 | 3 197–3 599 | 1 128–1 213 | 1 070–1 215 | 3 574–4 089 | 5 237–6 430 | 0.78–0.88 | +| 65 536 | 112–123 | 17 102–18 534 | 15 196–15 785 | 56 625–59 361 | 16 345–16 523 | 15 253–16 432 | 20 022–22 501 | 91 886–96 264 | 0.86–0.91 | + +What the numbers say: + +- **Per row, the target holds:** above ~4 K rows the expression costs under 1 ns/row and beats the scalar loop by 1.6–1.7×; the fold alone (`exec`) is ~0.27 ns/row at 64 K. + > ⊘ **Corrected by D-MPF-2 (§7):** "beats the scalar loop" compared against the `i128` filter-chain oracle, not an equal-work loop. A branch-free `i64` loop over the same `&[i32]` with a runtime threshold takes ~11 µs at 64 K rows (0.17 ns/row) and is ~3× faster than the SIMD-path fold. The per-row figure (<1 ns/row) stands; the comparison does not. +- **Per call at register scale, it does not:** at 64 rows (one `u64` word) one fold costs ~120 ns in mask-risc plus ~150 ns of dialect lowering, and the scalar loop (~80 ns) is faster than the whole expression (~600 ns). "Nanosecond hot execution" is therefore not met by the existing per-call API for one-word populations. +- `relalg::run` adds ~2.5 µs of fixed cost (normalize through the contract) over the dialect; worth it for rewrites, not for a hot one-word path. +- Materializing the mask (`keep+bind` vs `count+bind`) costs nothing measurable. +- 512-row figures vary 2× across runs; treat them as noisy. + +## 4. Gaps, each with its counterexample + +| id | gap | counterexample (test) | status | +|---|---|---|---| +| G1 | no element-wise value arithmetic (`x + y` per row) on the fold path | `lane_plus_lane_is_refused` | OPEN, owner decision: R2IL/mask-risc vs an existing ndarray kernel behind a typed method | +| G2 | no unsigned lane compare | `unsigned_compare_is_refused` | OPEN (known since D-SQLP-1) | +| G3 | no float lane kind; nothing IEEE is admitted | contract `Refusal::Ieee` | OPEN; floats route to ndarray kernels, measured, never certified | +| G4 | ~270 ns fixed cost per fold at one word | cost table row 64 | ATTRIBUTED (§7); no change made | + +No new primitive is proposed in this PR. G1 is the first place a counterexample shows composition is insufficient. + +## 5. Next stages (smallest experiment each; none authorized here) + +- **D-MPF-2:** attribute the 64-row fixed cost (Program build, scratch carve, tile loop, terminal) with counters; decide whether a one-word fast path is a mask-risc change or an existing `ndarray::simd` call. Gate: a one-word fold under 50 ns, equal to the oracle, or a recorded reason why not. +- **D-MPF-3 (G1):** the owner chooses where element-wise integer arithmetic lives. Counterexample to beat: `lane_plus_lane_is_refused`. +- **Stage 2 (ONNX wiring):** blocked until the ONNX/protobuf integration surface exists (§1) and S0 ownership is settled. +- **Stage 3 (DR64³):** the existing `relcomp::compose` R∘S chain is the start (#1747); separate plan. +- **Stage 4 (Palette256):** reuse BGZ/Helix/ndarray; the Burn compiled-linear identity defect (#1749/#1751 threads) must be fixed first. +- **Stage 5 (egglog):** one rewrite with a checkable proof, compiled to an existing recipe, fresh-frame rebind, oracle match, one refused rewrite. +- **Stage 6 (Optimum, #1751):** last. + +## 6. Open points + +| id | question | owner | status | +|---|---|---|---| +| MPF-OP-1 | where element-wise integer arithmetic lives (G1) | contract / R2IL owners | OPEN | +| MPF-OP-2 | which repository is the ONNX/protobuf fork | operator | OPEN | +| MPF-OP-3 | is a one-word fast path a mask-risc concern | mask-risc owner | OPEN (D-MPF-2) | +| MPF-OP-4 | S0 owner among #1744/#1745/#1750 | operator / EXO-50 | OPEN | + +## 7. D-MPF-2 — where the cost goes (measured, nothing changed) + +`examples/fixed_cost.rs`, `examples/floor_split.rs`, `examples/codegen_probe.rs` + `codegen-witness.sh`. Host: AVX-512 (`target-cpu=native`), 4 cores, other load present; two clean runs of three kept, one dropped as noisy. Every arm equals the `i128` oracle at every size before timing. No primitive, contract or ABI was changed. + +**Per call, one question (`Σ x > t`), each arm one layer more than the last** (ns, two runs): + +| rows | scalar `i64` | ndarray floor | mask-risc compiled | + recognition | + `Program::new` | dialect | +|---:|---:|---:|---:|---:|---:|---:| +| 64 | 12 | 39 | 94–99 | 97–100 | 113–125 | 171–172 | +| 4 096 | 664–686 | 1 721–1 831 | 1 789–1 901 | 1 792–1 911 | 1 811–1 936 | 1 839–1 975 | +| 65 536 | 10 967–11 817 | 30 884–42 348 | 31 048–31 051 | 30 578–31 190 | 32 925–36 005 | 33 765–36 822 | + +Allocations per call: 0 for the first four arms, 1 for `Program::new` and for the dialect (its `ops` vector is moved into each program and re-grown on the next fold). + +At 64 rows the 172 ns split as: ndarray floor 39 (memset, 4 compares, the bit walk), mask-risc execution ~57 (precheck, validation, tile dispatch, scratch carve), program build ~20 + 1 allocation, dialect ~50 (stack, epoch, op accumulation, result extraction). Recognition (`compile`) is ~3 ns. + +**Per row, the floor is the bottleneck, not mask-risc:** at 64 K rows mask-risc compiled ≈ ndarray floor. Inside the floor (`floor_split`, 65 536 rows, three runs): + +| selected | compare | `masked_sum_i32` | branch-free sum | scalar | +|---:|---:|---:|---:|---:| +| 0 | 5.4–9.2 µs | 1.2–1.7 µs | 12.2–20.8 µs | 10.8–11.5 µs | +| 32 429 | 5.5–6.0 µs | 24.6–26.5 µs | 12.2–12.5 µs | 10.7–10.9 µs | +| 65 536 | 5.2–5.6 µs | 43.8–52.6 µs | 12.2–17.7 µs | 11.1–12.4 µs | + +**LLVM** (`codegen-witness.sh`, PASS ×3): `gt_i32_to_mask` is one `vpcmpltd` + `kmov` per 16 lanes; `masked_sum_i32` is a scalar `tzcnt`/`blsr` walk with no vector register; the branch-free sum vectorizes (`vpsrlvq`, `vptestmq`, `vpmovsxdq`, `vpaddq`); the scalar row loop vectorizes on ymm. + +**Bottleneck:** `ndarray::simd::masked_sum_i32` walks set bits one at a time, so its cost tracks the popcount (0.03 → 0.67 ns/row from 0 % to 100 % selected) and it is ~75 % of the fold at half selectivity. + +**One proposed optimization (not made; owner of `ndarray::simd`):** choose the body by density. When a word has more than ~16 set bits, sum it branch-free; otherwise keep the walk. Negative controls already measured: the branch-free form equals the oracle at 0 %, 50 % and 100 %; its time is flat in selectivity (the mechanism is the missing branch, not the data); and the walk is 7–10× faster at 0 %, which is why the walk must stay for sparse words. Expected effect at 50 %: fold ~31 µs → ~18 µs at 64 K rows. Even then a one-pass fold stays slower than the scalar loop; the mask form pays off when the mask is reused or combined (NULL planes, several predicates), which this probe does not measure. + +**Fixed cost (G4):** the 50 ns gate is not met and is not reachable by tuning one layer: the floor alone is 39 ns at one word. A second proposal, for the r2il owner: keep the `ops` vector's capacity across folds (reclaim it from the previous program) to remove the one allocation per fold. Not made. + +## 8. Layer benchmark — ndarray primitives → mask-risc → R2IL polyfill + +> ⊘ The "R2IL" column below is the FoldDialect fold reading, re-run through its operand stack on every call. It is not a cost of R2IL routing; see §10. + +`examples/layers.rs`: five folds over `x > t` (t = 0, about half the rows), each through the three layers on the same resident `&[i32]`. Each layer's result is checked against the `i128` oracle at every size before it is timed. Runs 2 and 3 of three, ns per call (run 1 dropped as noisy at 64 rows). "Before r2il" is the mask-risc column, "after" is the R2IL column. + +| rows | op | scalar | ndarray | mask-risc | Δ | R2IL | Δ | allocs nd/mr/r2il | +|---:|---|---:|---:|---:|---:|---:|---:|---| +| 64 | count | 8–15 | 17–21 | 87–121 | +70–100 | 153–175 | +54–66 | 0/0/1 | +| 64 | exists | 6 | 17 | 90–95 | +73–78 | 149–172 | +59–77 | 0/0/1 | +| 64 | sum | 11–12 | 36–38 | 88–90 | +52 | 162–172 | +74–81 | 0/0/1 | +| 64 | min | 28–30 | 39–41 | 89–92 | +50–52 | 177 | +85–88 | 0/0/1 | +| 64 | max | 34–38 | 39–41 | 99–107 | +58–68 | 175–179 | +72–76 | 0/0/1 | +| 4 096 | count | 421–428 | 253–254 | 321–324 | +68–71 | 383–386 | +61 | 0/0/1 | +| 4 096 | exists | 6 | 255–280 | 317–349 | +62–69 | 381–385 | +36–64 | 0/0/1 | +| 4 096 | sum | 606–628 | 1 548–1 691 | 1 598–1 808 | +50–116 | 1 661–1 825 | +18–63 | 0/0/1 | +| 4 096 | min | 1 701–1 717 | 1 671–1 843 | 1 729–1 864 | +21–58 | 1 743–1 931 | +14–67 | 0/0/1 | +| 4 096 | max | 1 690–1 923 | 1 613–1 825 | 1 698–1 743 | −82–+85 | 1 803–1 935 | +60–236 | 0/0/1 | +| 65 536 | count | 9 386–9 850 | 4 809–5 367 | 4 988–5 380 | noise | 5 066 | noise | 0/0/1 | +| 65 536 | exists | 6 | 4 754–5 172 | 4 952–5 491 | noise | 5 341–5 540 | noise | 0/0/1 | +| 65 536 | sum | 9 924–10 951 | 28 658–30 789 | 28 661–30 575 | noise | 30 228–30 975 | noise | 0/0/1 | +| 65 536 | min | 28 676–30 353 | 32 996–33 760 | 31 903–32 749 | noise | 32 326–32 784 | noise | 0/0/1 | +| 65 536 | max | 27 833–30 056 | 29 384–32 385 | 34 665–34 770 | noise | 32 332–33 131 | noise | 0/0/1 | + +- **Per call:** mask-risc adds ~50–100 ns over ndarray, and the R2IL polyfill adds another ~55–90 ns plus one allocation. Both costs are fixed: at 4 096 rows they shrink to tens of ns, and at 65 536 rows they are inside run-to-run noise. The polyfill layers cost per call, not per row. +- **Per row:** count is the one fold where the ndarray path beats the scalar loop (~0.08 vs ~0.15 ns/row, `popcount_batch_u64`). For sum the ndarray path is ~3× slower than the scalar loop (the bit walk, §7). For min and max it matches the scalar loop, which does not vectorize them either (~0.45–0.5 ns/row). +- **Exists is not equal work:** the scalar loop stops at the first hit (row 4 here), so it takes 6 ns at every size. All three layers build the whole mask first. That is an algorithmic difference, not overhead: a fused predicate + `Any` that stops at the first non-empty tile would close it. Proposal only. + +## 9. D-MPF-2b (lab) — hybrid masked sum and mask reuse + +> ⊘ Commits `0834609fc` and `6efc2743e`, and this heading until now, called this D-MPF-3. That id is G1 (§5, `STATUS_BOARD.md`). This section is D-MPF-2b: it tests D-MPF-2's proposed optimization. + +`examples/hybrid.rs`, two runs, 65 536 rows unless stated, ns per call. Every arm is checked against an `i128` oracle at every configuration. Nothing in ndarray or R2IL was changed. A winning body is a proposal for the `ndarray::simd` owner, approved separately. + +**Hybrid sum.** Per mask word, three cases: +- an empty word is skipped; +- a word with at most `th` set bits uses the bit walk (the current `masked_sum_i32` body); +- a denser word is summed branch-free over all 64 lanes. + +Two mask shapes at the same density: `uniform` (independent bits) and `clustered` (whole words set or clear). + +| mask | density | ndarray walk | hybrid th=16 | branch-free only (th=0) | +|---|---:|---:|---:|---:| +| uniform | 1 % | 1 408–2 377 | 1 485–2 281 | 5 373–5 607 | +| uniform | 10 % | 12 124–12 325 | 5 538–5 891 | 11 450–12 762 | +| uniform | 50 % | 32 893–39 806 | 11 533–11 726 | 11 660–12 745 | +| uniform | 100 % | 42 732–44 840 | 11 708–12 334 | 12 130–12 477 | +| clustered | 10 % | 6 505–7 015 | 1 361–1 436 | 1 355–1 424 | +| clustered | 50 % | 25 445–25 887 | 5 424–6 925 | 5 852–6 763 | + +- `th = 16` matches or beats the walk at every density measured: ~2× at 10 %, ~3× at 50 %, ~3.7× at 100 % (uniform), 4–5× on clustered masks. +- Branch-free alone loses 2–4× on sparse uniform masks, so the walk has to stay for sparse words. +- The threshold is flat between 8 and 32, so it is not a sensitive knob. +- `masked_min_i32` / `masked_max_i32` use the same walk (`masked_fold_i32`, "cost proportional to the popcount") and are the next candidates. + +**Mask reuse across count, sum, min and max of `x > t`.** + +| rows | selected | recompute ×4 | reuse | reuse + hybrid(16) | fused scalar loop | +|---:|---:|---:|---:|---:|---:| +| 64 | 3 | 84–85 | 29–30 | 29–30 | 19–20 | +| 64 | 28 | 132–137 | 85–89 | 74–76 | 19–20 | +| 4 096 | 196 | 1 707–1 778 | 855–872 | 905–910 | 907–926 | +| 4 096 | 2 035 | 5 515–5 586 | 4 783–5 119 | 3 978–5 501 | 965–996 | +| 65 536 | 3 239 | 30 415–51 405 | 15 624–16 224 | 15 174–24 062 | 15 583–18 176 | +| 65 536 | 32 925 | 120 471–145 908 | 95 255–97 732 | 94 472–98 539 | 15 605–15 792 | + +- Reusing one mask removes the repeated compares: about 2× when sparse and ~20–30 % when dense. +- At ~5 % selected, the reused mask matches a fused scalar loop. At 50 % selected, the fused loop is ~6× faster, because min and max still walk every set bit. +- The mask form pays off when the selection is sparse or the reductions are bit-only (count). With dense selections and several value reductions, a fused pass wins until min and max get the same hybrid treatment. +- OPEN: hybrid min/max; a fused mask-then-multi-reduce terminal; the ndarray owner's call on adopting any body. + +## 10. D-MPF-4 — R2IL is a router: attribution, routing table, ClassId-prebound route + +### 10.1 Five things share the name; keep them apart + +| name | what it is (source) | executes? | +|---|---|---| +| r2sleigh `r2il` crate | P-code-derived IL types: `Varnode`, `SpaceId`, the 82-variant `R2ILOp`, `ArchSpec` (`r2sleigh/crates/r2il/src/lib.rs` crate doc) | no | +| r2sleigh `r2conc` / `r2sym` | the **machine reading**: `r2conc::SlabState::step`, op at a time over borrowed byte slabs, unsupported ops refused loudly; `r2sym`, symbolic with Z3 | yes, scalar machine state | +| `ogar-r2il` | proxy glue: the R2IL opcode table as an `ogar_loco::Vocabulary` (82 arities, no `r2sleigh` dependency), the fold-band opcodes, and `CallMask` + `project`. It defines two concept ids: `CONCEPT_R2IL_MACHINE` 0xC400 and `CONCEPT_R2IL_FOLD` 0xC401 (`OGAR/crates/ogar-r2il/src/lib.rs`) | no; a table | +| `lance-graph-r2il::FoldDialect` + `procs` | the **fold reading** (0xC401), "the fold reading of R2IL, executed over mask-risc" (its crate doc). An operand-stack dialect that builds a mask-risc `Program` at every scalar fold | yes, through mask-risc | +| `lance-graph-mask-risc` / `ndarray::simd` / `ndarray::simd_runtime` | program executor / kernels / per-op `LazyLock` trampolines for runtime dispatch (`simd_runtime/mod.rs` documents "~2-3 ns" per call) | yes | + +The §7 and §8 figures measured FoldDialect and mask-risc. They are not R2IL routing costs, and nothing on that path involves R2Sleigh. + +### 10.2 The earlier single-digit-ns figures, recovered + +None of them is an R2IL call. Each is the direct kernel that an R2IL route would bind to: + +| figure | exact symbol, work unit, condition | source | +|---|---|---| +| 2.2–2.8 ns | `ndarray::simd` AND, AND, popcount over one 4-word mask, against 54–70 ns for the same work as a `count_and(3)` invocation through r2il + mask-risc | `entries/2026-10-21-d-rft-1-cost-breakdown.md`; `GOLDSTAUB.md` G-2026-10-23-4 | +| 1.7–4.2 ns | `FacetCascade::shared_prefix_tiles`, one 8-tile point fold, all three arms | `plans/d-diamond-1-dual-fold-substrate-v1.md`, `STATUS_BOARD.md` D-DMD-P1 | +| 1.99–2.83 ns | `mature`, one ladder transition, median, 0 allocations | `entries/2026-10-11-maturation3-ladder-probe.md` | +| 1.9–2.3 ns | slot hit plus bit/word peek, hot | `entries/2026-09-25-aperture-masks-and-kernel-gap.md` | +| ~2–3 ns | `ndarray::simd_runtime`: per-call overhead of one indirect call plus a `LazyLock` acquire load (documented, not measured here) | `ndarray/src/simd_runtime/mod.rs` | + +Search was not exhaustive: the scout hit its 120-hit cap, and some spaces' hits were not listed. A direct grep for an ns figure next to r2il, dialect, r2conc or r2sym in lance-graph, OGAR and r2sleigh found nothing beyond §7/§8 and the D-NFR summary of them. That is "not found", not "absent". + +### 10.3 Routing table for the existing operation set + +Route classes: **DIRECT** = one existing native entrypoint, **COMPOSE** = several composed in Rust, **LOWER** = R2Sleigh's machine reading, **REFUSE** = no admitted method. "Before" means the check can run at bind or compile time and need not repeat per execution. + +| R2IL op / procedure (fold reading) | route | native entrypoint(s) | validation / certification | before? | +|---|---|---|---|---| +| `IntSLess` lane vs scalar, ± `IntNot` (`select_lt`, `filter_lt`, Gt/Lt/Ge/Le) | DIRECT | `gt/lt/ge/le_i32_to_mask` | lane is I32, row count | yes | +| `IntEqual` lane vs scalar (`select_u32`, `in_u32`) | DIRECT / COMPOSE | `eq/ne_{i32,u32,u64}_to_mask`, OR of several | lane kind | yes | +| `IntLess` / `IntLessEqual` (unsigned) | REFUSE today (`NoUnsignedLaneCompare`) | U64 lanes have `lt/le/gt/ge_u64_to_mask`; U32 lanes have only eq/ne | — | — | +| `IntAnd` / `IntOr` / `IntXor` / `IntNot` on masks (`intersect`, `union`, `differ`, `except`, `keep`) | DIRECT | `mask_and/or/xor/not/andnot` (+ `_assign`); fused mode `mask_ternlog::` | slot epoch (`StaleSlot`); program shape | yes, once the program exists | +| `PopCount` (`count`, `count_and`, `count_valid`) | DIRECT | `popcount_batch_u64`, `mask_ternlog_popcount` | — | — | +| `ANY` / `ALL` (`exists`, `all`, `not_exists`, `bool_or`, `bool_and`) | DIRECT | `mask_any`, `mask_all(n_rows)`, `mask_ternlog_any` | row count for `ALL` | yes | +| `SUM` (`sum`, `avg_parts`, `margin`) | COMPOSE | compare + `masked_sum_i32` (i64 accumulator) | — | — | +| `MIN` / `MAX` | COMPOSE | compare + `masked_min_i32` / `masked_max_i32`; empty → `None` | — | — | +| `IntAdd` / `IntSub` / `IntMult` / `IntSDiv` scalar × scalar | DIRECT (Rust `checked_*`) | none (scalar) | overflow and divide-by-zero refused | only for constant operands | +| `IntSub` lane − lane (G1, element-wise arithmetic) | REFUSE (`LaneVersusLane`) | partial: `masked_sum_wrapping_add_i32` (wrapping) | owner decision MPF-OP-1 | — | +| `VIA`, `GROUP_SUM`, `KEY_RUNS`, `RANGE`, `KEEP` (`group_sum`) | COMPOSE | mask-risc terminals; `eq_u32_via_to_mask` for predicates through a key | foreign-key and out-shape validation | the program-only half, yes (open) | +| `SCATTER_OR`, `BLEND` | REFUSE in the fold reading only | mask-risc has `Terminal::ScatterOrU32` / `BlendI32` | — | — | +| carry/borrow flags, `Load`/`Store` over address spaces, branches, `Subpiece`, width-exact machine integers | LOWER | `r2conc::SlabState::step` (concrete), `r2sym` (symbolic) | ArchSpec / space layout | yes | +| floats, atomics, `CallOther` | REFUSE in `r2conc` (`ConcError::Unsupported`); `r2sym` only | — | — | — | +| certification (Boolean / Distinct laws) | separate from execution | DR lab `relcomp` + egglog proofs | proof checked by egglog | always before; never per call | + +**Layers on the measured path that are not needed per call** (64 rows, §7). None of them is R2Sleigh. + +1. FoldDialect operand-stack interpretation, ~50 ns plus 1 allocation. Removed by an existing facility: run the dialect once and keep `last_program()` as a `CompiledProgram` (measured below). +2. `Program::new` plus recognition, ~20 ns plus 1 allocation. Removed by `CompiledProgram` (existing). +3. mask-risc `precheck` + `validate` on every `execute_compiled`, the ~50–60 ns remainder named in `entries/2026-09-25-tile-knee-and-compiled-lowering.md`. That entry's open proposal is to move the program-only half into `compile`. Not done. +4. The ONNX-shaped wrapper's attribute checks, per call. Diagnostic only. + +### 10.4 Measurement: ClassId → LazyLock → prebound callable + +`examples/classid_route.rs`. One `LazyLock` per ClassId resolves the route and its contract (op, dtype, shape, exact/approximate, capability). `bind` copies the callable into the binding. Every run checks frame, focus and scope with the DR lab's `FrameStore::check`. Nothing was added to a production crate. + +Two runs, `MPF_REPS=31`, ns per call (load ≈ 3 on 4 cores; the 4 096 and 65 536 rows vary ±15 % between runs): + +| rows | direct nd | route | route, no check | LazyLock + match per call | mr prepared | r2il prepared | dialect | onnx wrapper | +|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 64 | 36–39 | 61–67 | 36–39 | 37–39 | 85–94 | 85–91 | 167 | 204–219 | +| 512 | 185–203 | 199–223 | 185–203 | 186–192 | 229–249 | 229–250 | 358–381 | 344–395 | +| 65 536 | 30 024–30 534 | 29 678–30 771 | 27 590–30 518 | 27 847–30 137 | 30 223–42 385 | 28 425–40 277 | 30 570–38 039 | 30 448–31 990 | + +Allocations per call: 0 on every arm except the dialect (1) and the wrapper (3). + +- **Routing costs nothing measurable.** The prebound callable equals direct ndarray at every size. A per-call `LazyLock` deref plus `match` adds about 0–1 ns. +- **The remaining fixed cost is the guard, not the route:** the frame/focus/scope check alone costs 27–28 ns with 0 allocations. On its success path `FrameStore::check` executes two `lock cmpxchg` and two `lock decq` (objdump), because `Arc::downgrade` is called twice per check. +- **Prepared R2IL equals prepared mask-risc** (85–94 ns): FoldDialect's own program, kept as a `CompiledProgram`, is the same program. The gap to direct ndarray (~50 ns at one word) is mask-risc's per-call `precheck`/`validate` (§10.3 item 3). +- **Invariants asserted:** one resolution per ClassId (`[1, 1, 1, 1, 1, 1, 1]`); 0 route evaluations in the steady-state and multi-op loops; LOWER (`r2conc::SlabState::step(IntCarry)`) and REFUSE (f32 over an i32 lane) refused at bind; focus, scope and source changes refused at run; a binding sealed before `advance` refuses with `SourceChanged`; rebinding does not re-resolve the route. +- **Disable runs** (committed first, each red, then restored): `resolve` called in `run`, and `resolve` called in `bind`, each tripped "rebinding must not re-resolve the route" (4 vs 3). `resolve` called in the steady-state call shape tripped "one init per ClassId at most" (`SUM_GT` resolved 1 405 times). +- **LLVM** (`cargo rustc --release --example classid_route -- --emit=asm`): `probe_route_run` calls `FrameStore::check`, does two compares and then a tail `jmpq *80(%rax)` into the stored callable. It has no `LazyLock`, `Once` or backend `match`. `probe_lazy_per_call` reads the `Once` state, compares it, and calls through `ROUTES+216`, with `Once::call` only on the cold path. + +**One binding and one frame, four operations** (count, sum, min, any) at 4 096 rows. Rebinding per op: 4.5–4.7 µs. One binding, checked per run: 4.0–4.7 µs. Checked once: 4.1–5.4 µs. Direct ndarray with one shared mask and four consumers: 3.5–3.7 µs. The four route variants are within noise of each other. Direct ndarray is ahead because it computes the mask once and each prebound callable recomputes it. That gap belongs to mask reuse (§9), not routing. + +**One proposed optimization (not made; DR lab owner):** compare `Weak::as_ptr(&frame.store)` with `Arc::as_ptr(&self.identity)` (and the same for `seal`) instead of `Weak::ptr_eq(.., &Arc::downgrade(..))`. Identity semantics are unchanged: a live `Weak` keeps its allocation, so the pointer cannot be reused. The four atomic RMWs go away. Expected effect: most of the 27 ns. Unmeasured. + +OPEN: the mask-risc validation split (item 3); hybrid min/max; mask-sharing callables for multi-op bindings; the U64 unsigned-compare refusal (a mask-risc gap, since ndarray has the kernels). diff --git a/.github/workflows/math-polyfill-probe.yml b/.github/workflows/math-polyfill-probe.yml new file mode 100644 index 000000000..292d1732e --- /dev/null +++ b/.github/workflows/math-polyfill-probe.yml @@ -0,0 +1,70 @@ +name: Math Polyfill Probe (D-MPF-1) +# Own [workspace] excluded from root tests; a lab gate, not production +# admission. Mirrors egglog-r2il-probe.yml's toolchain setup: the pinned +# toolchain from rust-toolchain.toml, never `stable`. +on: + pull_request: + paths: + - "crates/math-polyfill-probe/**" + - "crates/lance-graph-contract/**" + - "crates/lance-graph-r2il/**" + - "crates/lance-graph-relalg/**" + - "crates/lance-graph-mask-risc/**" + - ".github/workflows/math-polyfill-probe.yml" + - "rust-toolchain.toml" + push: + branches: [main] + paths: + - "crates/math-polyfill-probe/**" + - "crates/lance-graph-contract/**" + - "crates/lance-graph-r2il/**" + - "crates/lance-graph-relalg/**" + - "crates/lance-graph-mask-risc/**" + - ".github/workflows/math-polyfill-probe.yml" + - "rust-toolchain.toml" + workflow_dispatch: +permissions: + contents: read +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true +env: + CARGO_TERM_COLOR: always + CARGO_INCREMENTAL: "0" + CARGO_PROFILE_DEV_DEBUG: "0" + CARGO_PROFILE_TEST_DEBUG: "0" + RUST_BACKTRACE: "1" +jobs: + isolated-probe: + name: excluded-crate fmt, clippy and tests + runs-on: ubuntu-24.04 + timeout-minutes: 60 + defaults: + run: + working-directory: lance-graph + steps: + - uses: actions/checkout@v4 + with: + path: lance-graph + persist-credentials: false + - name: Rust toolchain + run: rustup show + - name: Resolve target-cpu=native for this runner + id: cpu + run: | + flags=$(.github/scripts/resolve-native-rustflags.sh) + echo "RUSTFLAGS=$flags" >> "$GITHUB_ENV" + hash=$(printf %s "$flags" | sha256sum | cut -c1-16) + echo "flags=$hash" >> "$GITHUB_OUTPUT" + - uses: Swatinem/rust-cache@v2 + with: + key: cpu-${{ steps.cpu.outputs.flags }} + workspaces: lance-graph/crates/math-polyfill-probe + - name: fmt + run: cargo fmt --manifest-path crates/math-polyfill-probe/Cargo.toml --check + - name: clippy + run: cargo clippy --manifest-path crates/math-polyfill-probe/Cargo.toml --all-targets -- -D warnings + - name: tests + run: cargo test --release --manifest-path crates/math-polyfill-probe/Cargo.toml + - name: cost split (reported, not gated) + run: cargo run --release --example cost_split --manifest-path crates/math-polyfill-probe/Cargo.toml diff --git a/crates/lance-graph-r2il/src/lib.rs b/crates/lance-graph-r2il/src/lib.rs index 8d3b8f665..89560df35 100644 --- a/crates/lance-graph-r2il/src/lib.rs +++ b/crates/lance-graph-r2il/src/lib.rs @@ -1077,7 +1077,8 @@ impl Dialect for FoldDialect<'_> { let a = stack.pop().ok_or(FoldError::Underflow(f))?; match (a, b) { (Val::Scalar(x), Val::Scalar(y)) => { - stack.push(Val::Scalar(x - y)); + let r = x.checked_sub(y).ok_or(FoldError::Overflow(f))?; + stack.push(Val::Scalar(r)); Ok(()) } (Val::Address(_), Val::Address(_)) => Err(FoldError::LaneVersusLane(f)), diff --git a/crates/lance-graph-r2il/tests/dialect.rs b/crates/lance-graph-r2il/tests/dialect.rs index 743a9587d..996e4438a 100644 --- a/crates/lance-graph-r2il/tests/dialect.rs +++ b/crates/lance-graph-r2il/tests/dialect.rs @@ -164,6 +164,8 @@ fn every_named_refusal_is_still_produced() { (INT_S_DIV, i64::MIN, -1, FoldError::Overflow(INT_S_DIV)), (INT_ADD, i64::MAX, 1, FoldError::Overflow(INT_ADD)), (INT_MULT, i64::MAX, 2, FoldError::Overflow(INT_MULT)), + (INT_SUB, -2, i64::MAX, FoldError::Overflow(INT_SUB)), + (INT_SUB, i64::MIN, 1, FoldError::Overflow(INT_SUB)), ] { let mut d = FoldDialect::new(&planes, &foreign); let mut stack = vec![Val::Scalar(a), Val::Scalar(b)]; diff --git a/crates/math-polyfill-probe/Cargo.toml b/crates/math-polyfill-probe/Cargo.toml new file mode 100644 index 000000000..334cbb07f --- /dev/null +++ b/crates/math-polyfill-probe/Cargo.toml @@ -0,0 +1,30 @@ +[package] +name = "math-polyfill-probe" +version = "0.1.0" +edition = "2021" +publish = false +description = "D-MPF-1 lab probe: a small mathematical expression submitted through the existing R2IL procedures (lance-graph-r2il::procs) and relalg, lowered onto mask-risc, checked against an independent row-loop oracle, with typed refusals and a per-stage cost split. Adds no primitive, AST or executor." + +# Own workspace root, like lance-graph-relalg: it path-deps the r2il crate, +# which git-deps OGAR. Verify via +# `cargo test --release --manifest-path crates/math-polyfill-probe/Cargo.toml`. +[workspace] + +[dependencies] +lance-graph-mask-risc = { path = "../lance-graph-mask-risc" } +lance-graph-r2il = { path = "../lance-graph-r2il" } +lance-graph-relalg = { path = "../lance-graph-relalg" } +ogar-loco = { git = "https://github.com/AdaWorldAPI/OGAR", branch = "main" } + +# Examples only: the ndarray::simd floor the D-MPF-2 cost split compares +# against. Same source string as the copy mask-risc already pulls, so the +# lock keeps one ndarray. +[dev-dependencies] +ndarray = { git = "https://github.com/AdaWorldAPI/ndarray", branch = "master" } +# classid_route only: the DR lab FrameStore / Guards / Refusal, so the +# prebound route is checked by the existing frame-focus-scope validator +# instead of a stand-in. +dr-math-binding-probe = { path = "../dr-math-binding-probe" } + +[patch."https://github.com/AdaWorldAPI/lance-graph"] +lance-graph-contract = { path = "../lance-graph-contract" } diff --git a/crates/math-polyfill-probe/codegen-witness.sh b/crates/math-polyfill-probe/codegen-witness.sh new file mode 100755 index 000000000..518946399 --- /dev/null +++ b/crates/math-polyfill-probe/codegen-witness.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# D-MPF-2 LLVM witness for the primitives the native fold delegates to. +# Modelled on ndarray scripts/codegen-witness.sh: build the probe's assembly, +# run its self-check first, then assert per-symbol facts. Pinned observations +# of the current codegen, not targets: a change that moves them should fail +# here and be re-measured on purpose. +# +# Local, not in CI: the k-mask check assumes an AVX-512 build (the default +# config is target-cpu=native), and a CI runner's tier is not pinned. +# +# bash crates/math-polyfill-probe/codegen-witness.sh +# CARGO_ARGS='--config ../../.cargo/config-v3.toml' bash ... # pin a tier +set -euo pipefail +cd "$(dirname "$0")" +export CARGO_PROFILE_RELEASE_DEBUG=0 CARGO_INCREMENTAL=0 +# shellcheck disable=SC2086 +cargo ${CARGO_ARGS:-} run --release --example codegen_probe +rm -f target/release/examples/codegen_probe-*.s +# shellcheck disable=SC2086 +cargo ${CARGO_ARGS:-} rustc --release --example codegen_probe -- --emit=asm -C debuginfo=0 +asm=$(ls target/release/examples/codegen_probe-*.s) + +# Body of one probe symbol: from its label to its .size directive. +body() { awk -v s="$1" '$0 ~ "^_R[A-Za-z0-9_]*"s":$" {on=1; next} on && /\.size/ {exit} on' "$asm"; } +vec_lines() { grep -cE '%[xyz]mm|%k[0-7]' || true; } +count() { grep -cE "$1" || true; } + +fail=0 +check() { if eval "$2"; then echo "PASS: $1"; else echo "FAIL: $1"; fail=1; fi; } + +ms=$(body probe_masked_sum_i32) +bf=$(body probe_masked_sum_branchless) +gt=$(body probe_gt_i32_to_mask) +[ -n "$ms" ] && [ -n "$bf" ] && [ -n "$gt" ] || { echo "FAIL: probe symbol missing"; exit 1; } + +check "masked_sum_i32 is a scalar set-bit walk (tzcnt, no vector register)" \ + '[ "$(echo "$ms" | count "tzcnt|bsf")" -ge 1 ] && [ "$(echo "$ms" | vec_lines)" -eq 0 ]' +check "branch-free sum vectorizes (vector add, no tzcnt)" \ + '[ "$(echo "$bf" | count "vpaddq")" -ge 1 ] && [ "$(echo "$bf" | count "tzcnt|bsf")" -eq 0 ]' +check "gt_i32_to_mask compares 16 lanes into a k-mask" \ + '[ "$(echo "$gt" | count "vpcmp[a-z]*d")" -ge 1 ] && [ "$(echo "$gt" | count "%k[0-7]")" -ge 1 ]' +exit $fail diff --git a/crates/math-polyfill-probe/examples/classid_route.rs b/crates/math-polyfill-probe/examples/classid_route.rs new file mode 100644 index 000000000..3c5735ff1 --- /dev/null +++ b/crates/math-polyfill-probe/examples/classid_route.rs @@ -0,0 +1,642 @@ +//! D-MPF-4: ClassId → LazyLock → prebound native callable → direct kernel. +//! Lab probe, reported not gated. Adds no ABI, no interpreter, no dispatcher +//! to any production crate: everything below lives in this example. +//! +//! The question: what does R2IL routing cost when the route is resolved ONCE +//! per ClassId and the resolved callable is stored in the binding, against +//! calling the same `ndarray::simd` kernels directly? +//! +//! - One `LazyLock` per ClassId resolves the route (DIRECT / COMPOSE / +//! LOWER / REFUSE) and its contract. `INITS[c]` counts resolutions; +//! `ROUTE_EVALS` counts every route evaluation. +//! - `bind` reads the route once and copies the callable into [`Bound`]. +//! The hot path never touches the `LazyLock`, a registry, a `match` on +//! the backend, or a capability probe. +//! - Frame, focus and scope stay checked on every run, by the DR lab's own +//! `FrameStore::check` plus focus/scope equality. That check is +//! validation of a versioned identity, not route selection, and it is +//! timed as its own arm. +//! +//! Arms, all on the same `&[i32]`, `SUM { x : x > t }`: +//! direct nd `gt_i32_to_mask` + `masked_sum_i32` +//! route prebound callable + frame/focus/scope check +//! route nochk prebound callable alone (splits the check's cost) +//! lazy/call `LazyLock` deref + `match` per call (simd_runtime shape) +//! mr prepared mask-risc `execute_compiled` on a hand-built program +//! r2il prepared FoldDialect run ONCE, its `last_program` kept as a +//! `CompiledProgram`, then `execute_compiled` +//! dialect `SumWhere::eval` per call (the existing fold reading) +//! onnx wrapper diagnostic only; rebinds every call; returns i32 +//! Every arm is oracle-checked before timing. `MPF_REPS`, `MPF_ITERS`. + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::hint::black_box; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::LazyLock; +use std::time::Instant; + +use dr_math_binding_probe::{Focus, FrameRevisions, FrameStore, Guards, Refusal, Scope}; +use lance_graph_mask_risc::exec::{execute_compiled, tile_words_for, Scratch}; +use lance_graph_mask_risc::ir::{ + CompiledProgram, MaskOp, Operand, Pred as MPred, Program, Terminal, +}; +use lance_graph_mask_risc::{words_for, Foreign, LaneRef, Out, Planes, Value}; +use lance_graph_r2il::FoldDialect; +use math_polyfill_probe::native::{sum_where_oracle, Pred, SumWhere}; +use math_polyfill_probe::onnx_compat::reduce_sum_where_greater; +use ndarray::simd as nd; + +// ── counting allocator ───────────────────────────────────────────────────── + +struct Counting; +static ALLOCS: AtomicUsize = AtomicUsize::new(0); + +// SAFETY: forwards every call to `System` unchanged; only counts. +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, l: Layout) -> *mut u8 { + ALLOCS.fetch_add(1, Ordering::Relaxed); + // SAFETY: same contract as the caller's. + unsafe { System.alloc(l) } + } + unsafe fn dealloc(&self, p: *mut u8, l: Layout) { + // SAFETY: same contract as the caller's. + unsafe { System.dealloc(p, l) } + } +} + +#[global_allocator] +static A: Counting = Counting; + +// ── the route table ──────────────────────────────────────────────────────── + +/// A class of mathematical operation. The probe's ids are local; a real +/// router would key on the ClassView classid. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct ClassId(usize); + +const COUNT_GT: ClassId = ClassId(0); +const EXISTS_GT: ClassId = ClassId(1); +const SUM_GT: ClassId = ClassId(2); +const MIN_GT: ClassId = ClassId(3); +const SUM_OVER_MASK: ClassId = ClassId(4); +const INT_CARRY: ClassId = ClassId(5); +const F32_SUM_ON_I32: ClassId = ClassId(6); +const N_CLASSES: usize = 7; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Answer { + Count(u64), + Bool(bool), + Sum(i64), + Opt(Option), +} + +/// One native callable. `m` is caller-owned scratch, or for +/// [`SUM_OVER_MASK`] the resident mask itself. +type Kernel = fn(&[i32], i32, &mut [u64]) -> Answer; + +/// What a ClassId keeps through routing: op identity, dtype, shape, +/// exactness and the capability the route was resolved under. +#[derive(Clone, Copy, Debug)] +struct Contract { + op: &'static str, + dtype: &'static str, + shape: &'static str, + exact: bool, + caps: &'static str, +} + +#[derive(Clone, Copy, Debug)] +enum Route { + /// One existing native entrypoint. + Direct(Kernel), + /// Several existing native entrypoints, composed in Rust. + Compose(Kernel), + /// Needs R2Sleigh's machine reading; names its entrypoint. + Lower(&'static str), + /// No admitted native method. + Refuse(&'static str), +} + +#[derive(Debug)] +struct Resolved { + route: Route, + contract: Contract, +} + +static INITS: [AtomicUsize; N_CLASSES] = [const { AtomicUsize::new(0) }; N_CLASSES]; +static ROUTE_EVALS: AtomicUsize = AtomicUsize::new(0); + +fn caps() -> &'static str { + if cfg!(target_feature = "avx512f") { + "avx512f (compile-time)" + } else if cfg!(target_feature = "avx2") { + "avx2 (compile-time)" + } else { + "scalar or other (compile-time)" + } +} + +fn k_count(x: &[i32], t: i32, m: &mut [u64]) -> Answer { + nd::gt_i32_to_mask(x, t, m); + Answer::Count(nd::popcount_batch_u64(m)) +} +fn k_exists(x: &[i32], t: i32, m: &mut [u64]) -> Answer { + nd::gt_i32_to_mask(x, t, m); + Answer::Bool(nd::mask_any(m)) +} +fn k_sum(x: &[i32], t: i32, m: &mut [u64]) -> Answer { + nd::gt_i32_to_mask(x, t, m); + Answer::Sum(nd::masked_sum_i32(x, m)) +} +fn k_min(x: &[i32], t: i32, m: &mut [u64]) -> Answer { + nd::gt_i32_to_mask(x, t, m); + Answer::Opt(nd::masked_min_i32(x, m)) +} +fn k_sum_over_mask(x: &[i32], _t: i32, m: &mut [u64]) -> Answer { + Answer::Sum(nd::masked_sum_i32(x, m)) +} + +/// The one place a route is chosen. Runs once per ClassId, from its +/// `LazyLock`. +fn resolve(c: ClassId) -> Resolved { + INITS[c.0].fetch_add(1, Ordering::Relaxed); + ROUTE_EVALS.fetch_add(1, Ordering::Relaxed); + let exact = |op| Contract { + op, + dtype: "i32", + shape: "rank-1 lane", + exact: true, + caps: caps(), + }; + let (route, contract) = match c { + COUNT_GT => (Route::Compose(k_count), exact("count(x > t)")), + EXISTS_GT => (Route::Compose(k_exists), exact("exists(x > t)")), + SUM_GT => (Route::Compose(k_sum), exact("sum(x | x > t)")), + MIN_GT => (Route::Compose(k_min), exact("min(x | x > t)")), + SUM_OVER_MASK => ( + Route::Direct(k_sum_over_mask), + exact("sum(x | resident mask)"), + ), + INT_CARRY => ( + Route::Lower("r2conc::SlabState::step(IntCarry)"), + exact("carry-out of a machine-width add"), + ), + F32_SUM_ON_I32 => ( + Route::Refuse("f32 reduction requested over an i32 lane"), + Contract { + op: "sum_f32", + dtype: "f32 requested, i32 bound", + shape: "rank-1 lane", + exact: false, + caps: caps(), + }, + ), + _ => (Route::Refuse("unknown class"), exact("unknown")), + }; + Resolved { route, contract } +} + +static ROUTES: [LazyLock; N_CLASSES] = [ + LazyLock::new(|| resolve(ClassId(0))), + LazyLock::new(|| resolve(ClassId(1))), + LazyLock::new(|| resolve(ClassId(2))), + LazyLock::new(|| resolve(ClassId(3))), + LazyLock::new(|| resolve(ClassId(4))), + LazyLock::new(|| resolve(ClassId(5))), + LazyLock::new(|| resolve(ClassId(6))), +]; + +// ── binding ──────────────────────────────────────────────────────────────── + +#[derive(Debug, PartialEq)] +enum BindRefusal { + Lower(&'static str), + Refused(&'static str), + Guard(Refusal), +} + +/// A route bound to one lane under one frame, focus and scope. Holds the +/// callable itself, so running it never reads the route table. +struct Bound<'a> { + class: ClassId, + kernel: Kernel, + contract: Contract, + guards: Guards, + x: &'a [i32], + t: i32, +} + +fn bind<'a>( + class: ClassId, + store: &FrameStore, + guards: Guards, + x: &'a [i32], + t: i32, +) -> Result, BindRefusal> { + store.check(&guards.frame).map_err(BindRefusal::Guard)?; + let r = &*ROUTES[class.0]; + let kernel = match r.route { + Route::Direct(k) | Route::Compose(k) => k, + Route::Lower(e) => return Err(BindRefusal::Lower(e)), + Route::Refuse(why) => return Err(BindRefusal::Refused(why)), + }; + Ok(Bound { + class, + kernel, + contract: r.contract, + guards, + x, + t, + }) +} + +impl Bound<'_> { + /// Validate the versioned identity, then call the callable. + #[inline] + fn run( + &self, + store: &FrameStore, + focus: Focus, + scope: Scope, + m: &mut [u64], + ) -> Result { + store.check(&self.guards.frame)?; + if focus != self.guards.focus { + return Err(Refusal::FocusChanged); + } + if scope != self.guards.scope { + return Err(Refusal::ScopeChanged); + } + Ok((self.kernel)(self.x, self.t, m)) + } +} + +// The three steady-state call shapes, kept out of line so their assembly +// can be read on its own (`--emit=asm`, symbols `probe_*`). + +#[inline(never)] +pub fn probe_direct_sum(x: &[i32], t: i32, m: &mut [u64]) -> i64 { + nd::gt_i32_to_mask(x, t, m); + nd::masked_sum_i32(x, m) +} + +#[inline(never)] +fn probe_route_run( + b: &Bound<'_>, + store: &FrameStore, + focus: Focus, + scope: Scope, + m: &mut [u64], +) -> Result { + b.run(store, focus, scope, m) +} + +#[inline(never)] +fn probe_lazy_per_call(x: &[i32], t: i32, m: &mut [u64]) -> Answer { + match ROUTES[SUM_GT.0].route { + Route::Direct(k) | Route::Compose(k) => k(x, t, m), + _ => Answer::Bool(false), + } +} + +// ── measurement ──────────────────────────────────────────────────────────── + +fn env(name: &str, d: usize) -> usize { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(d) +} + +/// Median ns per call, plus heap allocations per call. +fn measure(reps: usize, iters: usize, mut f: impl FnMut()) -> (f64, f64) { + for _ in 0..100 { + f(); + } + let before = ALLOCS.load(Ordering::Relaxed); + for _ in 0..100 { + f(); + } + let allocs = (ALLOCS.load(Ordering::Relaxed) - before) as f64 / 100.0; + let mut v: Vec = (0..reps) + .map(|_| { + let t = Instant::now(); + for _ in 0..iters { + f(); + } + t.elapsed().as_nanos() as f64 / iters as f64 + }) + .collect(); + v.sort_by(f64::total_cmp); + (v[v.len() / 2], allocs) +} + +fn revs(source: u64) -> FrameRevisions { + FrameRevisions { + source, + relation_map: 1, + coordinates: 1, + } +} + +fn sum_of(v: Value) -> i64 { + match v { + Value::SumI64(s) => s, + other => panic!("MaskedSumI32 yields SumI64, got {other:?}"), + } +} + +fn main() { + let reps = env("MPF_REPS", 21); + let iters0 = env("MPF_ITERS", 4000); + let t: i32 = 0; + let focus = Focus { revision: 7 }; + let scope = Scope { id: 3 }; + let mut store = FrameStore::new(revs(1)); + let g = |s: &FrameStore| Guards { + frame: s.seal(), + focus, + scope, + }; + + // ── correctness across classes, before any timing ────────────────────── + let x: Vec = (0..1_000i32).map(|i| (i * 13) % 97 - 48).collect(); + let mut m = vec![0u64; words_for(x.len())]; + let sel: Vec = x.iter().copied().filter(|&v| v > t).collect(); + let want = [ + (COUNT_GT, Answer::Count(sel.len() as u64)), + (EXISTS_GT, Answer::Bool(!sel.is_empty())), + (SUM_GT, Answer::Sum(sel.iter().map(|&v| i64::from(v)).sum())), + (MIN_GT, Answer::Opt(sel.iter().copied().min())), + ]; + for (c, w) in want { + let b = bind(c, &store, g(&store), &x, t).expect("COMPOSE classes bind"); + assert_eq!(b.run(&store, focus, scope, &mut m), Ok(w), "{c:?}"); + // An empty selection: the contract's empty answers, not a guess. + let b = bind(c, &store, g(&store), &x, i32::MAX).unwrap(); + let empty = b.run(&store, focus, scope, &mut m).unwrap(); + assert!(matches!( + empty, + Answer::Count(0) | Answer::Bool(false) | Answer::Sum(0) | Answer::Opt(None) + )); + } + // DIRECT over a resident mask: the mask is the binding's input. + nd::gt_i32_to_mask(&x, t, &mut m); + let b = bind(SUM_OVER_MASK, &store, g(&store), &x, 0).unwrap(); + assert_eq!( + b.run(&store, focus, scope, &mut m), + Ok(Answer::Sum(sel.iter().map(|&v| i64::from(v)).sum())) + ); + // LOWER and REFUSE are refused at bind, never at run. + assert_eq!( + bind(INT_CARRY, &store, g(&store), &x, t).err(), + Some(BindRefusal::Lower("r2conc::SlabState::step(IntCarry)")) + ); + assert!(matches!( + bind(F32_SUM_ON_I32, &store, g(&store), &x, t).err(), + Some(BindRefusal::Refused(_)) + )); + + // ── refusal and rebinding on incompatible changes ────────────────────── + let b = bind(SUM_GT, &store, g(&store), &x, t).unwrap(); + assert_eq!( + b.run(&store, Focus { revision: 8 }, scope, &mut m), + Err(Refusal::FocusChanged) + ); + assert_eq!( + b.run(&store, focus, Scope { id: 4 }, &mut m), + Err(Refusal::ScopeChanged) + ); + drop(b); + let inits_before = INITS[SUM_GT.0].load(Ordering::Relaxed); + let stale_guards = g(&store); + store.advance(revs(2)); + assert_eq!( + bind(SUM_GT, &store, stale_guards.clone(), &x, t).err(), + Some(BindRefusal::Guard(Refusal::SourceChanged)) + ); + let b = bind(SUM_GT, &store, g(&store), &x, t).expect("rebind under the new frame"); + assert!(b.run(&store, focus, scope, &mut m).is_ok()); + assert_eq!( + INITS[SUM_GT.0].load(Ordering::Relaxed), + inits_before, + "rebinding must not re-resolve the route" + ); + let stale = Bound { + guards: stale_guards, + ..b + }; + assert_eq!( + stale.run(&store, focus, scope, &mut m), + Err(Refusal::SourceChanged), + "a binding sealed before the advance refuses at run" + ); + let c = stale.contract; + println!( + "contract kept through routing: {:?} → op {}, dtype {}, shape {}, exact {}, caps {}", + stale.class, c.op, c.dtype, c.shape, c.exact, c.caps + ); + + // ── steady state ─────────────────────────────────────────────────────── + let arms = [ + "direct nd", + "route", + "route nochk", + "lazy/call", + "mr prepared", + "r2il prepared", + "dialect", + "onnx wrapper", + ]; + print!("\n| rows |"); + for a in arms { + print!(" {a} |"); + } + println!(" allocs/call |"); + println!("|---:|{}---|", "---:|".repeat(arms.len())); + let evals_before = ROUTE_EVALS.load(Ordering::Relaxed); + for n in [64usize, 512, 4_096, 65_536] { + let iters = (iters0 * 64 / n).max(50); + let x: Vec = (0..n).map(|i| ((i * 13) % 97) as i32 - 48).collect(); + let mut m = vec![0u64; words_for(n)]; + let lanes = [LaneRef::I32(&x)]; + let planes = Planes { + n_rows: n, + masks: &[], + lanes: &lanes, + }; + let want = sum_where_oracle(&x, None, Pred::Gt(t)) as i64; + + let b = bind(SUM_GT, &store, g(&store), &x, t).unwrap(); + let cp = CompiledProgram::new(Program::new( + vec![MaskOp::Pred { + pred: MPred::GtI32 { lane: 0, t }, + under: None, + dst: 0, + }], + Terminal::MaskedSumI32 { + mask: Operand::Scratch(0), + lane: 0, + }, + )); + // The existing facility: run the fold reading once, keep its program. + let op = SumWhere { + lane: 0, + present: None, + pred: Pred::Gt(t), + }; + let mut d = FoldDialect::new(&planes, &Foreign::NONE); + let mut stack = Vec::with_capacity(8); + assert_eq!(op.eval(&mut d, &mut stack), Ok(want)); + let r2 = CompiledProgram::new(d.last_program().expect("one fold ran").clone()); + let mut buf = vec![0u64; tile_words_for(n) * 4]; + let run_cp = |cp: &CompiledProgram, buf: &mut [u64]| { + let mut s = Scratch::over_for_program(buf, cp.program(), n).unwrap(); + sum_of( + execute_compiled( + &cp.compiled(), + &planes, + &Foreign::NONE, + &mut s, + Out::None, + 0..n, + ) + .unwrap(), + ) + }; + + assert_eq!(probe_direct_sum(&x, t, &mut m), want); + assert_eq!( + probe_route_run(&b, &store, focus, scope, &mut m), + Ok(Answer::Sum(want)) + ); + assert_eq!((b.kernel)(&x, t, &mut m), Answer::Sum(want)); + assert_eq!(probe_lazy_per_call(&x, t, &mut m), Answer::Sum(want)); + assert_eq!(run_cp(&cp, &mut buf), want); + assert_eq!(run_cp(&r2, &mut buf), want); + let wrapped = + reduce_sum_where_greater(&x, &[n], t, &[], None, 0, 0).map(|o| i64::from(o.value)); + assert_eq!(wrapped, Ok(want)); + + let r = [ + measure(reps, iters, || { + black_box(probe_direct_sum(black_box(&x), black_box(t), &mut m)); + }), + measure(reps, iters, || { + let _ = black_box(probe_route_run(black_box(&b), &store, focus, scope, &mut m)); + }), + measure(reps, iters, || { + black_box((black_box(b.kernel))(black_box(&x), t, &mut m)); + }), + measure(reps, iters, || { + black_box(probe_lazy_per_call(black_box(&x), black_box(t), &mut m)); + }), + measure(reps, iters, || { + black_box(run_cp(&cp, &mut buf)); + }), + measure(reps, iters, || { + black_box(run_cp(&r2, &mut buf)); + }), + measure(reps, iters, || { + let _ = black_box(op.eval(&mut d, &mut stack)); + }), + measure(reps, iters, || { + let _ = black_box(reduce_sum_where_greater( + black_box(&x), + &[n], + black_box(t), + &[], + None, + 0, + 0, + )); + }), + ]; + print!("| {n} |"); + for (ns, _) in r { + print!(" {ns:.1} |"); + } + let a: Vec = r.iter().map(|(_, a)| format!("{a:.0}")).collect(); + println!(" {} |", a.join("/")); + } + let evals_after = ROUTE_EVALS.load(Ordering::Relaxed); + + // ── one binding, one frame, several operations ───────────────────────── + let n = 4_096; + let x: Vec = (0..n).map(|i| ((i * 13) % 97) as i32 - 48).collect(); + let mut m = vec![0u64; words_for(n)]; + let classes = [COUNT_GT, SUM_GT, MIN_GT, EXISTS_GT]; + let bound: Vec> = classes + .iter() + .map(|&c| bind(c, &store, g(&store), &x, t).unwrap()) + .collect(); + let iters = 2_000; + let multi = [ + ( + "rebind per op (4 binds + 4 runs)", + measure(reps, iters, || { + for &c in &classes { + let b = bind(c, &store, g(&store), &x, t).unwrap(); + black_box(b.run(&store, focus, scope, &mut m).unwrap()); + } + }), + ), + ( + "bind once, check per run (4 runs)", + measure(reps, iters, || { + for b in &bound { + black_box(b.run(&store, focus, scope, &mut m).unwrap()); + } + }), + ), + ( + "bind once, check once, 4 callables", + measure(reps, iters, || { + store.check(&bound[0].guards.frame).unwrap(); + for b in &bound { + black_box((b.kernel)(&x, t, &mut m)); + } + }), + ), + ( + "direct nd, one mask, 4 consumers", + measure(reps, iters, || { + nd::gt_i32_to_mask(&x, t, &mut m); + black_box(nd::popcount_batch_u64(&m)); + black_box(nd::masked_sum_i32(&x, &m)); + black_box(nd::masked_min_i32(&x, &m)); + black_box(nd::mask_any(&m)); + }), + ), + ]; + println!("\n| {n} rows, 4 ops | ns per answer set | allocs |"); + println!("|---|---:|---:|"); + for (name, (ns, a)) in multi { + println!("| {name} | {ns:.0} | {a:.0} |"); + } + let evals_end = ROUTE_EVALS.load(Ordering::Relaxed); + + // The run-time guard on its own: frame seal, focus, scope. + let (chk, chk_allocs) = measure(reps, 100_000, || { + let _ = black_box(store.check(black_box(&bound[0].guards.frame))); + black_box(black_box(focus) == bound[0].guards.focus); + black_box(black_box(scope) == bound[0].guards.scope); + }); + println!("\nframe/focus/scope check alone: {chk:.1} ns, {chk_allocs:.0} allocs"); + + let inits: Vec = INITS.iter().map(|a| a.load(Ordering::Relaxed)).collect(); + println!("\nresolutions per ClassId: {inits:?}"); + assert!( + inits.iter().all(|&i| i <= 1), + "one init per ClassId at most" + ); + println!( + "route evaluations during steady state: {} (single-op), {} (multi-op)", + evals_after - evals_before, + evals_end - evals_after + ); + assert_eq!(evals_after, evals_before); + assert_eq!(evals_end, evals_after); + println!("ns per call, median of MPF_REPS. Reported, not gated."); +} diff --git a/crates/math-polyfill-probe/examples/codegen_probe.rs b/crates/math-polyfill-probe/examples/codegen_probe.rs new file mode 100644 index 000000000..b0a61f292 --- /dev/null +++ b/crates/math-polyfill-probe/examples/codegen_probe.rs @@ -0,0 +1,61 @@ +//! D-MPF-2 LLVM witness: the ndarray primitives the native fold delegates to, +//! each behind one `#[inline(never)]` symbol so `--emit=asm` shows what they +//! compile to, after a self-check against a scalar reference. Modelled on +//! ndarray `scripts/codegen-witness.sh` + `examples/ternlog_codegen_probe.rs`. +//! +//! cargo rustc --release --example codegen_probe -- --emit=asm -C debuginfo=0 + +use std::hint::black_box; + +#[inline(never)] +pub fn probe_gt_i32_to_mask(x: &[i32], t: i32, m: &mut [u64]) { + ndarray::simd::gt_i32_to_mask(x, t, m); +} + +#[inline(never)] +pub fn probe_masked_sum_i32(x: &[i32], m: &[u64]) -> i64 { + ndarray::simd::masked_sum_i32(x, m) +} + +/// The proposed `masked_sum_i32` body: branch-free, one select per row, so +/// its cost should not depend on how many bits are set. +#[inline(never)] +pub fn probe_masked_sum_branchless(x: &[i32], m: &[u64]) -> i64 { + let mut acc = 0i64; + for (w, chunk) in x.chunks(64).enumerate() { + let word = m[w]; + let mut part = 0i64; + for (j, &v) in chunk.iter().enumerate() { + part += i64::from(v) & -(((word >> j) & 1) as i64); + } + acc += part; + } + acc +} + +#[inline(never)] +pub fn probe_scalar_sum_gt(x: &[i32], t: i32) -> i64 { + x.iter() + .map(|&v| if v > t { i64::from(v) } else { 0 }) + .sum() +} + +fn main() { + let x: Vec = (0..4096i32).map(|i| (i * 13) % 97 - 48).collect(); + let mut m = vec![0u64; 64]; + for t in [-100, -1, 0, 7, 100] { + probe_gt_i32_to_mask(black_box(&x), t, &mut m); + for (r, &v) in x.iter().enumerate() { + assert_eq!( + m[r / 64] >> (r % 64) & 1 == 1, + v > t, + "mask bit {r} at t={t}" + ); + } + let want: i64 = x.iter().filter(|&&v| v > t).map(|&v| i64::from(v)).sum(); + assert_eq!(probe_masked_sum_i32(black_box(&x), &m), want); + assert_eq!(probe_scalar_sum_gt(black_box(&x), t), want); + assert_eq!(probe_masked_sum_branchless(black_box(&x), &m), want); + } + println!("codegen_probe self-check: ok"); +} diff --git a/crates/math-polyfill-probe/examples/cost_split.rs b/crates/math-polyfill-probe/examples/cost_split.rs new file mode 100644 index 000000000..b5f8bf646 --- /dev/null +++ b/crates/math-polyfill-probe/examples/cost_split.rs @@ -0,0 +1,151 @@ +//! D-MPF-1 cost split, reported not gated. For each row count it times the +//! stages separately, each as the median over repetitions of a batch: +//! +//! - `bind`: `FoldDialect::new` (allocates the scratch once). +//! - `count`: selection + `COUNT` on a reused dialect (one fold). +//! - `expr`: the whole expression on a reused dialect (two folds + scalar ops). +//! - `count+bind` and `keep+bind`: a fresh dialect each time; their +//! difference is the cost of materializing the selection into a mask. +//! - `relalg`: `relalg::run` of the same selection count (normalize + lower + +//! one fold), fresh dialect inside. +//! - `exec`: mask-risc `execute_into` of the count program relalg lowered, +//! prebuilt, over a reused scratch buffer: the fold alone, no lowering. +//! - `oracle`: the scalar row loop. +//! +//! `MPF_REPS` (default 21) and `MPF_ITERS` (default 2000) set the sampling. + +use std::hint::black_box; +use std::time::Instant; + +use lance_graph_mask_risc::{execute_into, words_for, Foreign, LaneRef, Out, Planes, Scratch}; +use lance_graph_r2il::procs::{and_cmp_i32, count, is_not_null, keep, Order}; +use lance_graph_r2il::FoldDialect; +use lance_graph_relalg::{run, run_with_program, Cmp, Measure, Query, Rel}; +use math_polyfill_probe::{eval, oracle, Expr, Layout}; + +const L: Layout = Layout { + x: 0, + y: 1, + v: 2, + valid: 0, +}; +const E: Expr = Expr { + lo: -5, + hi: 20, + k: 3, +}; + +fn env(name: &str, d: usize) -> usize { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(d) +} + +/// Median ns per call over `reps` batches of `iters` calls. +fn time(reps: usize, iters: usize, mut f: impl FnMut()) -> f64 { + let mut v: Vec = (0..reps) + .map(|_| { + let t = Instant::now(); + for _ in 0..iters { + f(); + } + t.elapsed().as_nanos() as f64 / iters as f64 + }) + .collect(); + v.sort_by(f64::total_cmp); + v[v.len() / 2] +} + +fn main() { + let reps = env("MPF_REPS", 21); + let iters0 = env("MPF_ITERS", 2000); + println!( + "| rows | bind | exec | count | expr | count+bind | keep+bind | relalg | oracle | expr ns/row |" + ); + println!("|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|"); + for n in [64usize, 512, 4_096, 65_536] { + let iters = (iters0 * 64 / n).max(20); + let x: Vec = (0..n).map(|i| (i % 50) as i32 - 20).collect(); + let y: Vec = (0..n).map(|i| ((i * 7) % 31) as i32).collect(); + let v: Vec = (0..n).map(|i| ((i * 13) % 97) as i32 - 48).collect(); + let mut valid = vec![0u64; words_for(n)]; + for r in (0..n).filter(|r| r % 7 != 3) { + valid[r / 64] |= 1 << (r % 64); + } + let lanes = [LaneRef::I32(&x), LaneRef::I32(&y), LaneRef::I32(&v)]; + let masks: [&[u64]; 1] = [&valid]; + let p = Planes { + n_rows: n, + masks: &masks, + lanes: &lanes, + }; + + // Correctness first: a timing of a wrong answer is not a measurement. + let want = oracle(&x, &y, &v, &valid, E).1; + assert_eq!( + eval(&mut FoldDialect::new(&p, &Foreign::NONE), L, E).ok(), + want + ); + + let sel = |d: &mut FoldDialect<'_>, s: &mut Vec<_>| { + is_not_null(d, s, L.valid).unwrap(); + and_cmp_i32(d, s, L.x, Order::Ge, E.lo).unwrap(); + and_cmp_i32(d, s, L.y, Order::Lt, E.hi).unwrap(); + }; + let bind = time(reps, iters, || { + black_box(FoldDialect::new(black_box(&p), &Foreign::NONE)); + }); + let mut d = FoldDialect::new(&p, &Foreign::NONE); + let mut s = Vec::with_capacity(8); + let cnt = time(reps, iters, || { + sel(&mut d, &mut s); + count(&mut d, &mut s).unwrap(); + black_box(s.pop()); + }); + let expr = time(reps, iters, || { + black_box(eval(&mut d, L, E).unwrap()); + }); + let cnt_bind = time(reps, iters, || { + let mut d = FoldDialect::new(&p, &Foreign::NONE); + let mut s = Vec::with_capacity(8); + sel(&mut d, &mut s); + count(&mut d, &mut s).unwrap(); + black_box(s.pop()); + }); + let mut sink = vec![0u64; words_for(n)]; + let keep_bind = time(reps, iters, || { + let mut d = FoldDialect::with_keep(&p, &Foreign::NONE, &mut sink); + let mut s = Vec::with_capacity(8); + sel(&mut d, &mut s); + keep(&mut d, &mut s).unwrap(); + }); + black_box(&sink); + let q = Query { + rel: Rel::And(vec![ + Rel::Plane(L.valid), + Rel::i32(L.x, Cmp::Ge, E.lo), + Rel::i32(L.y, Cmp::Lt, E.hi), + ]), + measure: Measure::Count, + }; + let rel = time(reps, iters, || { + black_box(run(black_box(&q), &p, None).unwrap()); + }); + let (_, prog) = run_with_program(&q, &p, None).unwrap(); + let prog = prog.expect("the selection is not symbolic"); + let mut buf = vec![0u64; 1 << 16]; + let exec = time(reps, iters, || { + let mut sc = Scratch::over_for_program(&mut buf, &prog, n).unwrap(); + black_box(execute_into(&prog, &p, &Foreign::NONE, &mut sc, Out::None).unwrap()); + }); + let orc = time(reps, iters, || { + black_box(oracle(black_box(&x), &y, &v, &valid, E)); + }); + println!( + "| {n} | {bind:.0} | {exec:.0} | {cnt:.0} | {expr:.0} | {cnt_bind:.0} | {keep_bind:.0} | {rel:.0} | {orc:.0} | {:.2} |", + expr / n as f64 + ); + } + println!("\nAll figures ns per call (median). Reported, not gated."); +} diff --git a/crates/math-polyfill-probe/examples/fixed_cost.rs b/crates/math-polyfill-probe/examples/fixed_cost.rs new file mode 100644 index 000000000..71037275d --- /dev/null +++ b/crates/math-polyfill-probe/examples/fixed_cost.rs @@ -0,0 +1,215 @@ +//! D-MPF-2: where the fixed cost of one native fold goes. Reported, not gated. +//! +//! One question, `Σ { x[r] : x[r] > t }` over a resident `i32` lane, answered +//! by six arms that add one layer each, all on the same `&[i32]`: +//! +//! - `scalar`: a branch-free `i64` row loop (the compiler may vectorize it), +//! threshold passed through `black_box` so it is not specialised. +//! - `nd floor`: `ndarray::simd::gt_i32_to_mask` + `masked_sum_i32` into a +//! reused mask buffer, the two calls mask-risc delegates to. +//! - `mr compiled`: mask-risc `execute_compiled` on a program recognised once, +//! reused scratch. +//! - `mr into`: `execute_into` on a prebuilt program (adds recognition). +//! - `mr build`: `Program::new` + `Scratch::over` + `execute_into` per call, +//! what the dialect's fold boundary does. +//! - `dialect`: `SumWhere::eval` on a reused `FoldDialect`. +//! +//! Every arm is checked against an `i128` oracle at every size before timing. +//! Allocations per call are counted by a global allocator wrapper. +//! `MPF_REPS` (default 21) and `MPF_ITERS` (default 4000) set the sampling. + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::hint::black_box; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Instant; + +use lance_graph_mask_risc::exec::{execute_compiled, execute_into, tile_words_for, Scratch}; +use lance_graph_mask_risc::ir::{ + CompiledProgram, MaskOp, Operand, Pred as MPred, Program, Terminal, +}; +use lance_graph_mask_risc::{words_for, Foreign, LaneRef, Out, Planes, Value}; +use lance_graph_r2il::FoldDialect; +use math_polyfill_probe::native::{sum_where_oracle, Pred, SumWhere}; + +struct Counting; +static ALLOCS: AtomicUsize = AtomicUsize::new(0); + +// SAFETY: forwards every call to `System` unchanged; only counts. +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, l: Layout) -> *mut u8 { + ALLOCS.fetch_add(1, Ordering::Relaxed); + // SAFETY: same contract as the caller's. + unsafe { System.alloc(l) } + } + unsafe fn dealloc(&self, p: *mut u8, l: Layout) { + // SAFETY: same contract as the caller's. + unsafe { System.dealloc(p, l) } + } +} + +#[global_allocator] +static A: Counting = Counting; + +fn env(name: &str, d: usize) -> usize { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(d) +} + +/// Median ns per call, plus heap allocations per call. +fn measure(reps: usize, iters: usize, mut f: impl FnMut()) -> (f64, f64) { + let before = ALLOCS.load(Ordering::Relaxed); + for _ in 0..100 { + f(); + } + let allocs = (ALLOCS.load(Ordering::Relaxed) - before) as f64 / 100.0; + let mut v: Vec = (0..reps) + .map(|_| { + let t = Instant::now(); + for _ in 0..iters { + f(); + } + t.elapsed().as_nanos() as f64 / iters as f64 + }) + .collect(); + v.sort_by(f64::total_cmp); + (v[v.len() / 2], allocs) +} + +const T: i32 = 0; + +fn program() -> Program { + Program::new( + vec![MaskOp::Pred { + pred: MPred::GtI32 { lane: 0, t: T }, + under: None, + dst: 0, + }], + Terminal::MaskedSumI32 { + mask: Operand::Scratch(0), + lane: 0, + }, + ) +} + +fn sum_of(v: Value) -> i64 { + match v { + Value::SumI64(s) => s, + other => panic!("MaskedSumI32 yields SumI64, got {other:?}"), + } +} + +#[inline(never)] +fn scalar(x: &[i32], t: i32) -> i64 { + x.iter() + .map(|&v| if v > t { i64::from(v) } else { 0 }) + .sum() +} + +#[inline(never)] +fn nd_floor(x: &[i32], m: &mut [u64]) -> i64 { + ndarray::simd::gt_i32_to_mask(x, T, m); + ndarray::simd::masked_sum_i32(x, m) +} + +fn main() { + let reps = env("MPF_REPS", 21); + let iters0 = env("MPF_ITERS", 4000); + let op = SumWhere { + lane: 0, + present: None, + pred: Pred::Gt(T), + }; + let arms = [ + "scalar", + "nd floor", + "mr compiled", + "mr into", + "mr build", + "dialect", + ]; + print!("| rows |"); + for a in arms { + print!(" {a} |"); + } + println!(" allocs/call (per arm) |"); + println!("|---:|{}---|", "---:|".repeat(arms.len())); + for n in [64usize, 512, 4_096, 16_384, 65_536] { + let iters = (iters0 * 64 / n).max(50); + let x: Vec = (0..n).map(|i| ((i * 13) % 97) as i32 - 48).collect(); + let lanes = [LaneRef::I32(&x)]; + let planes = Planes { + n_rows: n, + masks: &[], + lanes: &lanes, + }; + let want = sum_where_oracle(&x, None, op.pred); + + let mut m = vec![0u64; words_for(n)]; + let prog = program(); + let cp = CompiledProgram::new(program()); + let mut buf = vec![0u64; tile_words_for(n) * 4]; + let mut d = FoldDialect::new(&planes, &Foreign::NONE); + let mut stack = Vec::with_capacity(8); + + // Equal work, checked: every arm answers the oracle at this size. + let run_compiled = |buf: &mut [u64]| { + let mut s = Scratch::over_for_program(buf, cp.program(), n).unwrap(); + sum_of( + execute_compiled( + &cp.compiled(), + &planes, + &Foreign::NONE, + &mut s, + Out::None, + 0..n, + ) + .unwrap(), + ) + }; + let run_into = |buf: &mut [u64]| { + let mut s = Scratch::over_for_program(buf, &prog, n).unwrap(); + sum_of(execute_into(&prog, &planes, &Foreign::NONE, &mut s, Out::None).unwrap()) + }; + let run_build = |buf: &mut [u64]| { + let p = program(); + let mut s = Scratch::over_for_program(buf, &p, n).unwrap(); + sum_of(execute_into(&p, &planes, &Foreign::NONE, &mut s, Out::None).unwrap()) + }; + assert_eq!(i128::from(scalar(&x, T)), want); + assert_eq!(i128::from(nd_floor(&x, &mut m)), want); + assert_eq!(i128::from(run_compiled(&mut buf)), want); + assert_eq!(i128::from(run_into(&mut buf)), want); + assert_eq!(i128::from(run_build(&mut buf)), want); + assert_eq!(op.eval(&mut d, &mut stack).map(i128::from), Ok(want)); + + let r = [ + measure(reps, iters, || { + black_box(scalar(black_box(&x), black_box(T))); + }), + measure(reps, iters, || { + black_box(nd_floor(black_box(&x), &mut m)); + }), + measure(reps, iters, || { + black_box(run_compiled(&mut buf)); + }), + measure(reps, iters, || { + black_box(run_into(&mut buf)); + }), + measure(reps, iters, || { + black_box(run_build(&mut buf)); + }), + measure(reps, iters, || { + black_box(op.eval(&mut d, &mut stack).unwrap()); + }), + ]; + print!("| {n} |"); + for (t, _) in r { + print!(" {t:.0} |"); + } + let a: Vec = r.iter().map(|(_, a)| format!("{a:.1}")).collect(); + println!(" {} |", a.join(" / ")); + } + println!("\nns per call, median of MPF_REPS. Reported, not gated."); +} diff --git a/crates/math-polyfill-probe/examples/floor_split.rs b/crates/math-polyfill-probe/examples/floor_split.rs new file mode 100644 index 000000000..8b40c7a0a --- /dev/null +++ b/crates/math-polyfill-probe/examples/floor_split.rs @@ -0,0 +1,76 @@ +//! D-MPF-2 follow-up: split the ndarray floor (`gt_i32_to_mask` then +//! `masked_sum_i32`) into its two calls and sweep selectivity. Reported, not +//! gated. Selectivity is set by the threshold over a lane in `[-48, 48]`: +//! `t = 100` selects nothing, `t = 0` about half, `t = -100` every row. + +use std::hint::black_box; +use std::time::Instant; + +fn med(reps: usize, iters: usize, mut f: impl FnMut()) -> f64 { + let mut v: Vec = (0..reps) + .map(|_| { + let t = Instant::now(); + for _ in 0..iters { + f(); + } + t.elapsed().as_nanos() as f64 / iters as f64 + }) + .collect(); + v.sort_by(f64::total_cmp); + v[v.len() / 2] +} + +/// The proposed `masked_sum_i32` body (see `codegen_probe`). +#[inline(never)] +fn masked_sum_branchless(x: &[i32], m: &[u64]) -> i64 { + let mut acc = 0i64; + for (w, chunk) in x.chunks(64).enumerate() { + let word = m[w]; + let mut part = 0i64; + for (j, &v) in chunk.iter().enumerate() { + part += i64::from(v) & -(((word >> j) & 1) as i64); + } + acc += part; + } + acc +} + +#[inline(never)] +fn scalar(x: &[i32], t: i32) -> i64 { + x.iter() + .map(|&v| if v > t { i64::from(v) } else { 0 }) + .sum() +} + +fn main() { + let n = 65_536; + let x: Vec = (0..n).map(|i| ((i * 13) % 97) as i32 - 48).collect(); + let mut m = vec![0u64; n / 64]; + println!("| t | selected | compare | masked sum | branch-free sum | scalar | ns/row compare | ns/row sum | ns/row branch-free |"); + println!("|---:|---:|---:|---:|---:|---:|---:|---:|---:|"); + for t in [100, 0, -100] { + ndarray::simd::gt_i32_to_mask(&x, t, &mut m); + let sel: u32 = m.iter().map(|w| w.count_ones()).sum(); + assert_eq!(ndarray::simd::masked_sum_i32(&x, &m), scalar(&x, t)); + let c = med(21, 200, || { + ndarray::simd::gt_i32_to_mask(black_box(&x), t, black_box(&mut m)) + }); + ndarray::simd::gt_i32_to_mask(&x, t, &mut m); + let s = med(21, 200, || { + black_box(ndarray::simd::masked_sum_i32(black_box(&x), black_box(&m))); + }); + assert_eq!(masked_sum_branchless(&x, &m), scalar(&x, t)); + let bf = med(21, 200, || { + black_box(masked_sum_branchless(black_box(&x), black_box(&m))); + }); + let sc = med(21, 200, || { + black_box(scalar(black_box(&x), black_box(t))); + }); + println!( + "| {t} | {sel} | {c:.0} | {s:.0} | {bf:.0} | {sc:.0} | {:.3} | {:.3} | {:.3} |", + c / n as f64, + s / n as f64, + bf / n as f64 + ); + } +} diff --git a/crates/math-polyfill-probe/examples/hybrid.rs b/crates/math-polyfill-probe/examples/hybrid.rs new file mode 100644 index 000000000..5e65f85ba --- /dev/null +++ b/crates/math-polyfill-probe/examples/hybrid.rs @@ -0,0 +1,250 @@ +//! D-MPF-2b (lab; first labelled D-MPF-3, which is G1): dense/sparse hybrid masked sum, and one mask reused across +//! several reductions. Reported, not gated. Nothing here changes ndarray or +//! R2IL; a winning body is a proposal for those owners. +//! +//! Part 1, the hybrid sum. Per 64-bit mask word: skip an empty word, walk the +//! set bits (`trailing_zeros`, the current `ndarray::simd::masked_sum_i32` +//! body) when the word holds at most `th` bits, else sum all 64 lanes +//! branch-free. `th = 64` is the walk alone, `th = 0` is branch-free alone. +//! Masks are drawn two ways at the same overall density: `uniform` (each bit +//! independent) and `clustered` (whole words set or clear), because a +//! per-word threshold can only help when density varies between words. +//! +//! Part 2, mask reuse. count, sum, min and max of `x > t`: +//! `recompute` builds the mask once per reduction, `reuse` builds it once, +//! `reuse+hybrid` also uses the hybrid sum, `fused scalar` does all four in +//! one row loop. Every arm is checked against an `i128` oracle before timing. + +use std::hint::black_box; +use std::time::Instant; + +use ndarray::simd::{ + gt_i32_to_mask, masked_max_i32, masked_min_i32, masked_sum_i32, popcount_batch_u64, +}; + +fn med(reps: usize, iters: usize, mut f: impl FnMut() -> i64) -> f64 { + let mut v: Vec = (0..reps) + .map(|_| { + let t = Instant::now(); + for _ in 0..iters { + black_box(f()); + } + t.elapsed().as_nanos() as f64 / iters as f64 + }) + .collect(); + v.sort_by(f64::total_cmp); + v[v.len() / 2] +} + +/// SplitMix64: deterministic, no dependency. +struct Rng(u64); +impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + /// True with probability `p_permille / 1000`. + fn hit(&mut self, p_permille: u64) -> bool { + self.next() % 1000 < p_permille + } +} + +fn mask_uniform(n: usize, p: u64, r: &mut Rng) -> Vec { + let mut m = vec![0u64; n.div_ceil(64)]; + for i in 0..n { + if r.hit(p) { + m[i / 64] |= 1 << (i % 64); + } + } + m +} + +fn mask_clustered(n: usize, p: u64, r: &mut Rng) -> Vec { + (0..n.div_ceil(64)) + .map(|_| if r.hit(p) { u64::MAX } else { 0 }) + .collect() +} + +fn oracle_sum(x: &[i32], m: &[u64]) -> i64 { + let s: i128 = (0..x.len()) + .filter(|&i| m[i / 64] >> (i % 64) & 1 == 1) + .map(|i| i128::from(x[i])) + .sum(); + i64::try_from(s).unwrap() +} + +/// The proposed body. `x.len()` must be a multiple of 64 (true for every size +/// used here); the tail case is the existing walk's business. +#[inline(never)] +fn hybrid_sum(x: &[i32], m: &[u64], th: u32) -> i64 { + let mut acc = 0i64; + let (chunks, _) = x.as_chunks::<64>(); + for (w, chunk) in chunks.iter().enumerate() { + let word = m[w]; + let c = word.count_ones(); + if c == 0 { + continue; + } + if c > th { + let mut part = 0i64; + for (j, &v) in chunk.iter().enumerate() { + part += i64::from(v) & -(((word >> j) & 1) as i64); + } + acc += part; + } else { + let mut bits = word; + while bits != 0 { + acc += i64::from(chunk[bits.trailing_zeros() as usize]); + bits &= bits - 1; + } + } + } + acc +} + +#[derive(Debug, PartialEq, Eq, Clone, Copy)] +struct Four { + count: i64, + sum: i64, + min: Option, + max: Option, +} + +fn four_oracle(x: &[i32], t: i32) -> Four { + let sel: Vec = x.iter().copied().filter(|&v| v > t).collect(); + Four { + count: sel.len() as i64, + sum: i64::try_from(sel.iter().map(|&v| i128::from(v)).sum::()).unwrap(), + min: sel.iter().copied().min(), + max: sel.iter().copied().max(), + } +} + +#[inline(never)] +fn four_recompute(x: &[i32], t: i32, m: &mut [u64]) -> Four { + gt_i32_to_mask(x, t, m); + let count = popcount_batch_u64(m) as i64; + gt_i32_to_mask(x, t, m); + let sum = masked_sum_i32(x, m); + gt_i32_to_mask(x, t, m); + let min = masked_min_i32(x, m); + gt_i32_to_mask(x, t, m); + let max = masked_max_i32(x, m); + Four { + count, + sum, + min, + max, + } +} + +#[inline(never)] +fn four_reuse(x: &[i32], t: i32, m: &mut [u64], hybrid: Option) -> Four { + gt_i32_to_mask(x, t, m); + Four { + count: popcount_batch_u64(m) as i64, + sum: match hybrid { + Some(th) => hybrid_sum(x, m, th), + None => masked_sum_i32(x, m), + }, + min: masked_min_i32(x, m), + max: masked_max_i32(x, m), + } +} + +#[inline(never)] +fn four_fused_scalar(x: &[i32], t: i32) -> Four { + let (mut count, mut sum, mut min, mut max) = (0i64, 0i64, i32::MAX, i32::MIN); + for &v in x { + let s = v > t; + count += i64::from(s); + sum += if s { i64::from(v) } else { 0 }; + min = if s && v < min { v } else { min }; + max = if s && v > max { v } else { max }; + } + Four { + count, + sum, + min: (count > 0).then_some(min), + max: (count > 0).then_some(max), + } +} + +fn pack(f: Four) -> i64 { + f.count ^ f.sum ^ i64::from(f.min.unwrap_or(0)) ^ i64::from(f.max.unwrap_or(0)) +} + +fn main() { + let reps = 21; + let n = 65_536; + let mut r = Rng(0x5EED); + let x: Vec = (0..n).map(|_| (r.next() % 2001) as i32 - 1000).collect(); + + println!("## Part 1: hybrid masked sum, {n} rows, ns per call (median of {reps})\n"); + let ths = [64u32, 48, 32, 16, 8, 0]; + print!("| mask | density | walk (ndarray) |"); + for th in ths { + print!(" th={th} |"); + } + println!(); + println!("|---|---:|---:|{}", "---:|".repeat(ths.len())); + for (kind, gen) in [ + ( + "uniform", + mask_uniform as fn(usize, u64, &mut Rng) -> Vec, + ), + ("clustered", mask_clustered), + ] { + for p in [0u64, 10, 100, 250, 500, 750, 1000] { + let m = gen(n, p, &mut r); + let want = oracle_sum(&x, &m); + assert_eq!(masked_sum_i32(&x, &m), want); + let walk = med(reps, 200, || masked_sum_i32(black_box(&x), black_box(&m))); + print!("| {kind} | {:.1} % | {walk:.0} |", p as f64 / 10.0); + for th in ths { + assert_eq!(hybrid_sum(&x, &m, th), want, "{kind} p={p} th={th}"); + let t = med(reps, 200, || hybrid_sum(black_box(&x), black_box(&m), th)); + print!(" {t:.0} |"); + } + println!(); + } + } + + println!( + "\n## Part 2: one mask, four reductions (count, sum, min, max of x > t), ns per call\n" + ); + println!("| rows | t | selected | recompute ×4 | reuse | reuse + hybrid(16) | fused scalar |"); + println!("|---:|---:|---:|---:|---:|---:|---:|"); + for nn in [64usize, 4_096, 65_536] { + let xs = &x[..nn]; + let mut m = vec![0u64; nn.div_ceil(64)]; + let iters = (200 * 65_536 / nn).max(200); + for t in [900, 0, -900] { + let want = four_oracle(xs, t); + assert_eq!(four_recompute(xs, t, &mut m), want); + assert_eq!(four_reuse(xs, t, &mut m, None), want); + assert_eq!(four_reuse(xs, t, &mut m, Some(16)), want); + assert_eq!(four_fused_scalar(xs, t), want); + let a = med(reps, iters, || { + pack(four_recompute(black_box(xs), black_box(t), &mut m)) + }); + let b = med(reps, iters, || { + pack(four_reuse(black_box(xs), black_box(t), &mut m, None)) + }); + let c = med(reps, iters, || { + pack(four_reuse(black_box(xs), black_box(t), &mut m, Some(16))) + }); + let d = med(reps, iters, || { + pack(four_fused_scalar(black_box(xs), black_box(t))) + }); + println!( + "| {nn} | {t} | {} | {a:.0} | {b:.0} | {c:.0} | {d:.0} |", + want.count + ); + } + } + println!("\nReported, not gated."); +} diff --git a/crates/math-polyfill-probe/examples/layers.rs b/crates/math-polyfill-probe/examples/layers.rs new file mode 100644 index 000000000..348e39bee --- /dev/null +++ b/crates/math-polyfill-probe/examples/layers.rs @@ -0,0 +1,246 @@ +//! The layer benchmark: every op through ndarray primitives, then the +//! lance-graph mask-risc program, then the R2IL polyfill, on the same +//! resident `&[i32]` lane. Reported, not gated. +//! +//! Each op is ` { x[r] : x[r] > t }` for fold in count, exists, sum, +//! min, max. Layers: +//! +//! - `ndarray`: `gt_i32_to_mask` into a reused mask, then the matching +//! reducer (`popcount_batch_u64`, `mask_any`, `masked_sum_i32`, +//! `masked_min_i32`, `masked_max_i32`). +//! - `mask-risc`: one `Program` (`Pred::GtI32` + the matching terminal), +//! recognised once, run with `execute_compiled` over reused scratch. +//! - `r2il`: the same question lowered onto a reused `FoldDialect` with R2IL +//! calls (`LOAD`, `IntSLess`) and the `procs` fold (`count`, `exists`, +//! `sum`, `min`, `max`). +//! +//! "before r2il" is the mask-risc column, "after" the r2il column; `Δ` is +//! each layer minus the one below. Every layer is checked against an `i128` +//! oracle at every size before timing. `scalar` is an equal-work row loop +//! for reference. `MPF_REPS` (default 31), `MPF_ITERS` (default 4000). + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::hint::black_box; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Instant; + +use lance_graph_mask_risc::exec::{execute_compiled, tile_words_for, Scratch}; +use lance_graph_mask_risc::ir::{CompiledProgram, MaskOp, Operand, Pred, Program, Terminal}; +use lance_graph_mask_risc::{words_for, Foreign, LaneRef, Out, Planes, Value}; +use lance_graph_r2il::procs; +use lance_graph_r2il::{FoldDialect, FoldError, Val, INT_S_LESS, LOAD}; +use ogar_loco::Dialect; + +struct Counting; +static ALLOCS: AtomicUsize = AtomicUsize::new(0); + +// SAFETY: forwards every call to `System` unchanged; only counts. +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, l: Layout) -> *mut u8 { + ALLOCS.fetch_add(1, Ordering::Relaxed); + // SAFETY: same contract as the caller's. + unsafe { System.alloc(l) } + } + unsafe fn dealloc(&self, p: *mut u8, l: Layout) { + // SAFETY: same contract as the caller's. + unsafe { System.dealloc(p, l) } + } +} + +#[global_allocator] +static A: Counting = Counting; + +fn env(name: &str, d: usize) -> usize { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(d) +} + +/// Median ns per call and heap allocations per call. +fn measure(reps: usize, iters: usize, mut f: impl FnMut() -> Option) -> (f64, f64) { + let before = ALLOCS.load(Ordering::Relaxed); + for _ in 0..100 { + black_box(f()); + } + let allocs = (ALLOCS.load(Ordering::Relaxed) - before) as f64 / 100.0; + let mut v: Vec = (0..reps) + .map(|_| { + let t = Instant::now(); + for _ in 0..iters { + black_box(f()); + } + t.elapsed().as_nanos() as f64 / iters as f64 + }) + .collect(); + v.sort_by(f64::total_cmp); + (v[v.len() / 2], allocs) +} + +#[derive(Clone, Copy, Debug)] +enum Op { + Count, + Exists, + Sum, + Min, + Max, +} + +const OPS: [Op; 5] = [Op::Count, Op::Exists, Op::Sum, Op::Min, Op::Max]; + +/// The answer every layer must give. `None` is an empty MIN/MAX. +fn oracle(x: &[i32], t: i32, op: Op) -> Option { + let sel = x.iter().filter(|&&v| v > t).map(|&v| i128::from(v)); + let v = match op { + Op::Count => Some(sel.count() as i128), + Op::Exists => Some(i128::from(x.iter().any(|&v| v > t))), + Op::Sum => Some(sel.sum()), + Op::Min => sel.min(), + Op::Max => sel.max(), + }; + v.map(|v| i64::try_from(v).expect("fits")) +} + +#[inline(never)] +fn scalar(x: &[i32], t: i32, op: Op) -> Option { + let it = x.iter().copied(); + match op { + Op::Count => Some(it.filter(|&v| v > t).count() as i64), + Op::Exists => Some(i64::from(it.clone().any(|v| v > t))), + Op::Sum => Some(it.map(|v| if v > t { i64::from(v) } else { 0 }).sum()), + Op::Min => it.filter(|&v| v > t).min().map(i64::from), + Op::Max => it.filter(|&v| v > t).max().map(i64::from), + } +} + +#[inline(never)] +fn nd(x: &[i32], t: i32, op: Op, m: &mut [u64]) -> Option { + use ndarray::simd::{ + gt_i32_to_mask, mask_any, masked_max_i32, masked_min_i32, masked_sum_i32, + popcount_batch_u64, + }; + gt_i32_to_mask(x, t, m); + match op { + Op::Count => Some(popcount_batch_u64(m) as i64), + Op::Exists => Some(i64::from(mask_any(m))), + Op::Sum => Some(masked_sum_i32(x, m)), + Op::Min => masked_min_i32(x, m).map(i64::from), + Op::Max => masked_max_i32(x, m).map(i64::from), + } +} + +fn program(t: i32, op: Op) -> Program { + let mask = Operand::Scratch(0); + let terminal = match op { + Op::Count => Terminal::Count { mask }, + Op::Exists => Terminal::Any { mask }, + Op::Sum => Terminal::MaskedSumI32 { mask, lane: 0 }, + Op::Min => Terminal::MaskedMinI32 { mask, lane: 0 }, + Op::Max => Terminal::MaskedMaxI32 { mask, lane: 0 }, + }; + Program::new( + vec![MaskOp::Pred { + pred: Pred::GtI32 { lane: 0, t }, + under: None, + dst: 0, + }], + terminal, + ) +} + +fn value(v: Value) -> Option { + match v { + Value::Count(n) => Some(n as i64), + Value::Bool(b) => Some(i64::from(b)), + Value::SumI64(s) => Some(s), + Value::OptI32(o) => o.map(i64::from), + other => panic!("scalar terminal, got {other:?}"), + } +} + +/// `x > t` as `t < x` (`IntSLess` is the only ordered compare), then the fold. +fn r2il( + d: &mut FoldDialect<'_>, + s: &mut Vec, + t: i32, + op: Op, +) -> Result, FoldError> { + s.clear(); + s.push(Val::Scalar(i64::from(t))); + s.push(Val::Scalar(0)); + d.call(LOAD, [1, 0, 0], s)?; + d.call(INT_S_LESS, [0; 3], s)?; + match op { + Op::Count => procs::count(d, s)?, + Op::Exists => procs::exists(d, s)?, + Op::Sum => procs::sum(d, s, 0)?, + Op::Min => procs::min(d, s, 0)?, + Op::Max => procs::max(d, s, 0)?, + } + let v = match s.pop() { + Some(Val::Scalar(v)) => v, + _ => return Err(FoldError::WrongOperandKind(INT_S_LESS)), + }; + // `procs::min`/`max` push i64::MAX / i64::MIN for an empty mask. + Ok(match (op, v) { + (Op::Min, i64::MAX) | (Op::Max, i64::MIN) => None, + _ => Some(v), + }) +} + +fn main() { + let reps = env("MPF_REPS", 31); + let iters0 = env("MPF_ITERS", 4000); + let t = 0; + println!("| rows | op | scalar | ndarray | mask-risc (before r2il) | Δ mask-risc | r2il (after) | Δ r2il | allocs nd/mr/r2il |"); + println!("|---:|---|---:|---:|---:|---:|---:|---:|---|"); + for n in [64usize, 4_096, 65_536] { + let iters = (iters0 * 64 / n).max(50); + let x: Vec = (0..n).map(|i| ((i * 13) % 97) as i32 - 48).collect(); + let lanes = [LaneRef::I32(&x)]; + let planes = Planes { + n_rows: n, + masks: &[], + lanes: &lanes, + }; + let mut m = vec![0u64; words_for(n)]; + let mut buf = vec![0u64; tile_words_for(n) * 4]; + let mut d = FoldDialect::new(&planes, &Foreign::NONE); + let mut s = Vec::with_capacity(8); + for op in OPS { + let want = oracle(&x, t, op); + let cp = CompiledProgram::new(program(t, op)); + let mr = |buf: &mut [u64]| { + let mut sc = Scratch::over_for_program(buf, cp.program(), n).unwrap(); + value( + execute_compiled( + &cp.compiled(), + &planes, + &Foreign::NONE, + &mut sc, + Out::None, + 0..n, + ) + .unwrap(), + ) + }; + assert_eq!(scalar(&x, t, op), want, "scalar {op:?} n={n}"); + assert_eq!(nd(&x, t, op, &mut m), want, "ndarray {op:?} n={n}"); + assert_eq!(mr(&mut buf), want, "mask-risc {op:?} n={n}"); + assert_eq!(r2il(&mut d, &mut s, t, op), Ok(want), "r2il {op:?} n={n}"); + + let (sc, _) = measure(reps, iters, || scalar(black_box(&x), black_box(t), op)); + let (a, aa) = measure(reps, iters, || nd(black_box(&x), black_box(t), op, &mut m)); + let (b, ba) = measure(reps, iters, || mr(&mut buf)); + let (c, ca) = measure(reps, iters, || { + r2il(&mut d, &mut s, black_box(t), op).unwrap() + }); + println!( + "| {n} | {op:?} | {sc:.0} | {a:.0} | {b:.0} | {:+.0} | {c:.0} | {:+.0} | {aa:.0}/{ba:.0}/{ca:.0} |", + b - a, + c - b + ); + } + } + println!("\nns per call, median of MPF_REPS; t = 0 selects about half the rows. Reported, not gated."); +} diff --git a/crates/math-polyfill-probe/examples/native_op_cost.rs b/crates/math-polyfill-probe/examples/native_op_cost.rs new file mode 100644 index 000000000..5e3210293 --- /dev/null +++ b/crates/math-polyfill-probe/examples/native_op_cost.rs @@ -0,0 +1,119 @@ +//! D-MPF-1b cost split for the native `SumWhere` op, reported not gated. +//! +//! - `bind`: build `Planes` and `FoldDialect::new` (the setup a caller pays +//! once per population). +//! - `hot`: `SumWhere::eval` on a reused dialect and a reused stack. +//! - `hot allocs`: heap allocations per hot call, counted by a global +//! allocator wrapper (the hot path's allocation budget). +//! - `onnx wrapper`: `reduce_sum_where_greater`, which binds on every call. +//! - `oracle`: the `i128` row loop. +//! +//! `MPF_REPS` (default 21) and `MPF_ITERS` (default 2000) set the sampling. + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::hint::black_box; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Instant; + +use lance_graph_mask_risc::{Foreign, LaneRef, Planes}; +use lance_graph_r2il::FoldDialect; +use math_polyfill_probe::native::{sum_where_oracle, Pred, SumWhere}; +use math_polyfill_probe::onnx_compat::reduce_sum_where_greater; + +struct Counting; +static ALLOCS: AtomicUsize = AtomicUsize::new(0); + +// SAFETY: forwards every call to `System` unchanged; only counts. +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, l: Layout) -> *mut u8 { + ALLOCS.fetch_add(1, Ordering::Relaxed); + // SAFETY: same contract as the caller's. + unsafe { System.alloc(l) } + } + unsafe fn dealloc(&self, p: *mut u8, l: Layout) { + // SAFETY: same contract as the caller's. + unsafe { System.dealloc(p, l) } + } +} + +#[global_allocator] +static A: Counting = Counting; + +fn env(name: &str, d: usize) -> usize { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(d) +} + +fn time(reps: usize, iters: usize, mut f: impl FnMut()) -> f64 { + let mut v: Vec = (0..reps) + .map(|_| { + let t = Instant::now(); + for _ in 0..iters { + f(); + } + t.elapsed().as_nanos() as f64 / iters as f64 + }) + .collect(); + v.sort_by(f64::total_cmp); + v[v.len() / 2] +} + +fn main() { + let reps = env("MPF_REPS", 21); + let iters0 = env("MPF_ITERS", 2000); + let op = SumWhere { + lane: 0, + present: None, + pred: Pred::Gt(0), + }; + println!("| rows | bind | hot | hot allocs | onnx wrapper | oracle | hot ns/row |"); + println!("|---:|---:|---:|---:|---:|---:|---:|"); + for n in [64usize, 4_096, 65_536] { + let iters = (iters0 * 64 / n).max(20); + let x: Vec = (0..n).map(|i| ((i * 13) % 97) as i32 - 48).collect(); + let lanes = [LaneRef::I32(&x)]; + let planes = Planes { + n_rows: n, + masks: &[], + lanes: &lanes, + }; + + let want = sum_where_oracle(&x, None, op.pred); + let mut d = FoldDialect::new(&planes, &Foreign::NONE); + let mut s = Vec::with_capacity(8); + assert_eq!(op.eval(&mut d, &mut s).map(i128::from), Ok(want)); + let w = reduce_sum_where_greater(&x, &[n], 0, &[], None, 0, 0).unwrap(); + assert_eq!(i128::from(w.value), want); + + let bind = time(reps, iters, || { + let lanes = [LaneRef::I32(black_box(&x))]; + let p = Planes { + n_rows: n, + masks: &[], + lanes: &lanes, + }; + black_box(FoldDialect::new(&p, &Foreign::NONE)); + }); + let hot = time(reps, iters, || { + black_box(op.eval(&mut d, &mut s).unwrap()); + }); + let before = ALLOCS.load(Ordering::Relaxed); + for _ in 0..100 { + black_box(op.eval(&mut d, &mut s).unwrap()); + } + let allocs = (ALLOCS.load(Ordering::Relaxed) - before) as f64 / 100.0; + let wrap = time(reps, iters, || { + black_box(reduce_sum_where_greater(black_box(&x), &[n], 0, &[], None, 0, 0).unwrap()); + }); + let orc = time(reps, iters, || { + black_box(sum_where_oracle(black_box(&x), None, op.pred)); + }); + println!( + "| {n} | {bind:.0} | {hot:.0} | {allocs:.1} | {wrap:.0} | {orc:.0} | {:.2} |", + hot / n as f64 + ); + } + println!("\nns per call, median. Reported, not gated."); +} diff --git a/crates/math-polyfill-probe/src/lib.rs b/crates/math-polyfill-probe/src/lib.rs new file mode 100644 index 000000000..56a415ba2 --- /dev/null +++ b/crates/math-polyfill-probe/src/lib.rs @@ -0,0 +1,101 @@ +//! D-MPF-1: the mathematical polyfill, as it already exists. +//! +//! The Stage 1 polyfill API is not new code. It is the typed procedure layer +//! over R2IL that already ships in [`lance_graph_r2il::procs`] (selection, +//! comparison, NULL presence, `COUNT` / `SUM` / `MIN` / `MAX`, Boolean set +//! operations) plus the checked scalar arithmetic of +//! [`FoldDialect`] (`IntAdd`, `IntSub`, `IntMult`, `IntSDiv`). A procedure +//! emits R2IL calls only; mask-risc executes them. This crate adds no opcode, +//! no IR, no executor. +//! +//! What this crate holds: +//! +//! - [`Expr`]: one small expression, written ONLY with those procedures: +//! `SUM(v | sel) − k·COUNT(sel)` with +//! `sel = valid(v) ∧ x ≥ lo ∧ y < hi`. +//! - [`eval`]: lowers it through the procedures onto a lent [`FoldDialect`]. +//! - [`oracle`]: an independent row loop over the same lanes, in `i128`, that +//! never touches the dialect, mask-risc or the procedures. +//! +//! Semantics preserved, and where each is decided: +//! +//! - Domain: `i32` lanes, `i64` results. Integer only; there is no float lane +//! in the fold path, so no IEEE claim is made. +//! - NULL: `v`'s presence plane. A row whose `v` is NULL is outside `sel` +//! (SQL `WHERE` drops UNKNOWN); it is not counted and not summed. +//! - Multiplicity: rows are a set of row positions; `COUNT` is the popcount +//! of the selection mask. +//! - Overflow: refused, never wrapped (`FoldError::Overflow`). +//! - Exactness: the answer is exact, and must equal [`oracle`] bit for bit. + +pub mod native; +pub mod onnx_compat; + +use lance_graph_r2il::procs::{and_cmp_i32, count, is_not_null, sum, Order}; +use lance_graph_r2il::{FoldDialect, FoldError, Val, INT_MULT, INT_SUB}; +use ogar_loco::Dialect; + +/// Lane and plane indexes the expression reads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Layout { + /// `I32` lane `x`. + pub x: u16, + /// `I32` lane `y`. + pub y: u16, + /// `I32` lane `v`, the summed value. + pub v: u16, + /// Resident mask plane: rows where `v` is present (not NULL). + pub valid: u16, +} + +/// `SUM(v | sel) − k·COUNT(sel)`, `sel = valid(v) ∧ x ≥ lo ∧ y < hi`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Expr { + /// Lower bound on `x`, inclusive. + pub lo: i32, + /// Upper bound on `y`, exclusive. + pub hi: i32, + /// The scalar multiplier on the count. + pub k: i64, +} + +/// Push the selection mask. Emitted once per fold: a fold ends the epoch of +/// every slot built before it (see `procs::margin`). +fn select(d: &mut FoldDialect<'_>, s: &mut Vec, l: Layout, e: Expr) -> Result<(), FoldError> { + is_not_null(d, s, l.valid)?; + and_cmp_i32(d, s, l.x, Order::Ge, e.lo)?; + and_cmp_i32(d, s, l.y, Order::Lt, e.hi) +} + +/// Evaluate `e` on the dialect's planes. Every step is an existing procedure +/// or an existing scalar R2IL call. +pub fn eval(d: &mut FoldDialect<'_>, l: Layout, e: Expr) -> Result { + let mut s = Vec::with_capacity(8); + select(d, &mut s, l, e)?; + sum(d, &mut s, l.v)?; + select(d, &mut s, l, e)?; + count(d, &mut s)?; + s.push(Val::Scalar(e.k)); + d.call(INT_MULT, [0; 3], &mut s)?; + d.call(INT_SUB, [0; 3], &mut s)?; + match s.pop() { + Some(Val::Scalar(v)) if s.is_empty() => Ok(v), + _ => Err(FoldError::WrongOperandKind(INT_SUB)), + } +} + +/// The rows the selection keeps, and the answer, computed row by row. +/// `None` when the exact answer does not fit `i64` (the dialect must refuse). +pub fn oracle(x: &[i32], y: &[i32], v: &[i32], valid: &[u64], e: Expr) -> (u64, Option) { + let mut sum: i128 = 0; + let mut n: u64 = 0; + for r in 0..v.len() { + let present = valid[r / 64] >> (r % 64) & 1 == 1; + if present && x[r] >= e.lo && y[r] < e.hi { + sum += i128::from(v[r]); + n += 1; + } + } + let ans = sum - i128::from(e.k) * i128::from(n); + (n, i64::try_from(ans).ok()) +} diff --git a/crates/math-polyfill-probe/src/native.rs b/crates/math-polyfill-probe/src/native.rs new file mode 100644 index 000000000..49128a57f --- /dev/null +++ b/crates/math-polyfill-probe/src/native.rs @@ -0,0 +1,101 @@ +//! One native polyfill operation, with no ONNX type anywhere in it. +//! +//! [`SumWhere`]: `Σ { x[r] : x present at r ∧ x[r] ⋈ t }`, exact, in `i64`. +//! Inputs are a lane index and an optional presence plane into the caller's +//! [`Planes`](lance_graph_mask_risc::Planes); the operation is lowered onto a +//! lent [`FoldDialect`] with existing R2IL calls only (`LOAD`, `IntSLess`, +//! `IntNot`, `IntAnd`, `SUM`) and executed by mask-risc. +//! +//! Binding (building `Planes` and the dialect) is the caller's, so a hot loop +//! reuses one dialect and pays only for the fold. + +use lance_graph_r2il::procs::{is_not_null, sum}; +use lance_graph_r2il::{FoldDialect, FoldError, Val, INT_AND, INT_NOT, INT_S_LESS, LOAD}; +use ogar_loco::Dialect; + +/// `LOAD` space of an `I32` lane, as `FoldDialect` reads it. +const SPACE_I32: u8 = 1; + +/// The comparison of each element against the threshold. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Pred { + /// `x > t` + Gt(i32), + /// `x >= t` + Ge(i32), + /// `x < t` + Lt(i32), + /// `x <= t` + Le(i32), +} + +impl Pred { + /// The predicate on one value: the reference the oracle uses. + pub fn holds(self, x: i32) -> bool { + match self { + Pred::Gt(t) => x > t, + Pred::Ge(t) => x >= t, + Pred::Lt(t) => x < t, + Pred::Le(t) => x <= t, + } + } +} + +/// `Σ { lane[r] : present(r) ∧ pred(lane[r]) }`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct SumWhere { + /// `I32` lane index. + pub lane: u16, + /// Presence plane of the lane, if it has NULLs. + pub present: Option, + /// Which elements count. + pub pred: Pred, +} + +impl SumWhere { + /// Lower and run. `stack` is caller-owned so a hot loop allocates nothing. + pub fn eval(self, d: &mut FoldDialect<'_>, stack: &mut Vec) -> Result { + stack.clear(); + // `IntSLess` is the only ordered compare: `x < t` and `t < x`, and the + // inclusive forms are their negations. + let (t, lane_first, negate) = match self.pred { + Pred::Lt(t) => (t, true, false), + Pred::Ge(t) => (t, true, true), + Pred::Gt(t) => (t, false, false), + Pred::Le(t) => (t, false, true), + }; + let t = Val::Scalar(i64::from(t)); + let lane = Val::Scalar(i64::from(self.lane)); + if lane_first { + stack.push(lane); + d.call(LOAD, [SPACE_I32, 0, 0], stack)?; + stack.push(t); + } else { + stack.push(t); + stack.push(lane); + d.call(LOAD, [SPACE_I32, 0, 0], stack)?; + } + d.call(INT_S_LESS, [0; 3], stack)?; + if negate { + d.call(INT_NOT, [0; 3], stack)?; + } + if let Some(p) = self.present { + is_not_null(d, stack, p)?; + d.call(INT_AND, [0; 3], stack)?; + } + sum(d, stack, self.lane)?; + match stack.pop() { + Some(Val::Scalar(v)) if stack.is_empty() => Ok(v), + _ => Err(FoldError::WrongOperandKind(INT_AND)), + } + } +} + +/// The reference: a row loop in `i128`, sharing no code with [`SumWhere`]. +pub fn sum_where_oracle(x: &[i32], present: Option<&[u64]>, pred: Pred) -> i128 { + (0..x.len()) + .filter(|&r| present.is_none_or(|p| p[r / 64] >> (r % 64) & 1 == 1)) + .filter(|&r| pred.holds(x[r])) + .map(|r| i128::from(x[r])) + .sum() +} diff --git a/crates/math-polyfill-probe/src/onnx_compat.rs b/crates/math-polyfill-probe/src/onnx_compat.rs new file mode 100644 index 000000000..1722441fd --- /dev/null +++ b/crates/math-polyfill-probe/src/onnx_compat.rs @@ -0,0 +1,97 @@ +//! ONNX semantic conformance for one composition, WITHOUT an ONNX graph. +//! +//! [`reduce_sum_where_greater`] answers what the ONNX subgraph +//! `ReduceSum(Where(Greater(X, T), X, 0))` answers, for the admitted subset +//! below, by calling the native [`SumWhere`](crate::native::SumWhere). There is +//! no `ModelProto`, no node, no tensor object and no runtime: the operator +//! attributes arrive as plain Rust values and are checked against the ONNX +//! specification (opset 18+ `ReduceSum`, where `axes` is an input). +//! +//! Admitted subset (everything else is a typed [`Refusal`]): +//! - `X`: `int32`, rank 1. `T`: `int32`, a scalar (shape `[]` or `[1]`; +//! multidirectional broadcasting of a scalar). +//! - `axes`: absent, or exactly `[0]` / `[-1]`. With absent axes and +//! `noop_with_empty_axes = 0`, ONNX reduces every axis, which for rank 1 is +//! the same reduction. +//! - `keepdims`: `0` (output shape `[]`) or `1` (output shape `[1]`). +//! - Output dtype is `int32`, as ONNX `ReduceSum` keeps the input type. The +//! specification does not define int32 overflow; this wrapper refuses an +//! exact sum outside `i32` instead of wrapping. + +use lance_graph_mask_risc::{Foreign, LaneRef, Planes}; +use lance_graph_r2il::{FoldDialect, FoldError}; + +use crate::native::{Pred, SumWhere}; + +/// Why the wrapper does not answer. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Refusal { + /// `X` is not rank 1. + Rank(usize), + /// `T` is not a scalar. + ThresholdNotScalar(Vec), + /// `axes` other than absent, `[0]` or `[-1]`. + Axes(Vec), + /// `keepdims` other than 0 or 1. + KeepDims(i64), + /// `noop_with_empty_axes = 1` with absent axes returns `X` unchanged, a + /// tensor result outside this wrapper's scalar output. + NoopWithEmptyAxes, + /// The exact sum does not fit `int32`. + OutOfDtypeRange(i64), + /// The native polyfill refused. + Native(FoldError), +} + +/// The output, shaped as ONNX shapes it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Output { + /// The value. + pub value: i32, + /// `true`: shape `[1]` (`keepdims = 1`); `false`: shape `[]`. + pub keepdims: bool, +} + +/// `ReduceSum(Where(Greater(X, T), X, 0))` for the admitted subset. +pub fn reduce_sum_where_greater( + x: &[i32], + x_shape: &[usize], + t: i32, + t_shape: &[usize], + axes: Option<&[i64]>, + keepdims: i64, + noop_with_empty_axes: i64, +) -> Result { + if x_shape.len() != 1 || x_shape[0] != x.len() { + return Err(Refusal::Rank(x_shape.len())); + } + if !(t_shape.is_empty() || t_shape == [1]) { + return Err(Refusal::ThresholdNotScalar(t_shape.to_vec())); + } + match axes { + None if noop_with_empty_axes == 1 => return Err(Refusal::NoopWithEmptyAxes), + None => {} + Some(a) if a == [0] || a == [-1] => {} + Some(a) => return Err(Refusal::Axes(a.to_vec())), + } + let keepdims = match keepdims { + 0 => false, + 1 => true, + k => return Err(Refusal::KeepDims(k)), + }; + let lanes = [LaneRef::I32(x)]; + let planes = Planes { + n_rows: x.len(), + masks: &[], + lanes: &lanes, + }; + let mut d = FoldDialect::new(&planes, &Foreign::NONE); + let op = SumWhere { + lane: 0, + present: None, + pred: Pred::Gt(t), + }; + let s = op.eval(&mut d, &mut Vec::new()).map_err(Refusal::Native)?; + let value = i32::try_from(s).map_err(|_| Refusal::OutOfDtypeRange(s))?; + Ok(Output { value, keepdims }) +} diff --git a/crates/math-polyfill-probe/tests/native_op.rs b/crates/math-polyfill-probe/tests/native_op.rs new file mode 100644 index 000000000..a1dd38a95 --- /dev/null +++ b/crates/math-polyfill-probe/tests/native_op.rs @@ -0,0 +1,178 @@ +//! D-MPF-1b: one native op (`SumWhere`) and its ONNX-semantics wrapper. +//! Every answer is compared with `sum_where_oracle` (an `i128` row loop) or +//! with ONNX's own definition written out by hand. + +use lance_graph_mask_risc::{words_for, Foreign, LaneRef, Planes}; +use lance_graph_r2il::{FoldDialect, FoldError}; +use math_polyfill_probe::native::{sum_where_oracle, Pred, SumWhere}; +use math_polyfill_probe::onnx_compat::{reduce_sum_where_greater, Output, Refusal}; + +fn lane(n: usize) -> Vec { + (0..n).map(|i| ((i * 13) % 97) as i32 - 48).collect() +} + +fn plane(n: usize) -> Vec { + let mut w = vec![0u64; words_for(n)]; + for r in (0..n).filter(|r| r % 7 != 3) { + w[r / 64] |= 1 << (r % 64); + } + w +} + +const PREDS: [Pred; 8] = [ + Pred::Gt(0), + Pred::Ge(0), + Pred::Lt(0), + Pred::Le(0), + Pred::Gt(100), // empty + Pred::Lt(-100), // empty + Pred::Ge(-48), // everything + Pred::Le(-48), // one value class +]; + +/// FAILS IF: any predicate form, the NULL plane, or the sum diverges from the +/// row loop, at one word, a ragged tail and several tiles; on one reused +/// dialect, so a stale-slot bug across calls would show. +#[test] +fn sum_where_equals_the_oracle_on_a_reused_dialect() { + for n in [64, 1_000, 65_537] { + let x = lane(n); + let p = plane(n); + let lanes = [LaneRef::I32(&x)]; + let masks: [&[u64]; 1] = [&p]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &lanes, + }; + let mut d = FoldDialect::new(&planes, &Foreign::NONE); + let mut s = Vec::new(); + for pred in PREDS { + for present in [None, Some(0u16)] { + let got = SumWhere { + lane: 0, + present, + pred, + } + .eval(&mut d, &mut s); + let want = sum_where_oracle(&x, present.map(|_| p.as_slice()), pred); + assert_eq!(got.map(i128::from), Ok(want), "n={n} {pred:?} {present:?}"); + } + } + } +} + +/// FAILS IF: the fixture is vacuous: some predicate must select nothing, some +/// a strict subset, and the NULL plane must change at least one answer. +#[test] +fn the_fixture_is_not_vacuous() { + let x = lane(1_000); + let p = plane(1_000); + assert_eq!(sum_where_oracle(&x, None, Pred::Gt(100)), 0); + let all: i128 = x.iter().map(|&v| i128::from(v)).sum(); + let part = sum_where_oracle(&x, None, Pred::Gt(0)); + assert!(part != 0 && part != all); + assert_ne!( + sum_where_oracle(&x, None, Pred::Gt(0)), + sum_where_oracle(&x, Some(&p), Pred::Gt(0)) + ); +} + +/// ONNX's definition, written out: Greater → bool, Where(c, X, 0), ReduceSum. +fn onnx_reference(x: &[i32], t: i32) -> i64 { + let cond: Vec = x.iter().map(|&v| v > t).collect(); + let picked: Vec = x + .iter() + .zip(&cond) + .map(|(&v, &c)| if c { i64::from(v) } else { 0 }) + .collect(); + picked.iter().sum() +} + +/// FAILS IF: the wrapper's answer differs from ONNX's definition for any +/// admitted attribute combination. +#[test] +fn the_wrapper_matches_the_onnx_definition() { + let x = lane(1_000); + for t in [-100, -1, 0, 7, 100] { + let want = i32::try_from(onnx_reference(&x, t)).unwrap(); + for (axes, kd) in [ + (None, 0), + (Some(&[0i64][..]), 0), + (Some(&[-1][..]), 1), + (None, 1), + ] { + for t_shape in [&[][..], &[1][..]] { + let got = reduce_sum_where_greater(&x, &[1_000], t, t_shape, axes, kd, 0); + assert_eq!( + got, + Ok(Output { + value: want, + keepdims: kd == 1 + }), + "t={t} {axes:?} {kd}" + ); + } + } + } +} + +/// FAILS IF: an out-of-subset call is answered instead of refused. +#[test] +fn out_of_subset_calls_are_refused() { + let x = lane(8); + let r = |shape: &[usize], ts: &[usize], axes: Option<&[i64]>, kd: i64, noop: i64| { + reduce_sum_where_greater(&x, shape, 0, ts, axes, kd, noop) + }; + assert_eq!(r(&[2, 4], &[], None, 0, 0), Err(Refusal::Rank(2))); + assert_eq!( + r(&[8], &[8], None, 0, 0), + Err(Refusal::ThresholdNotScalar(vec![8])) + ); + assert_eq!(r(&[8], &[], Some(&[1]), 0, 0), Err(Refusal::Axes(vec![1]))); + assert_eq!(r(&[8], &[], None, 2, 0), Err(Refusal::KeepDims(2))); + assert_eq!(r(&[8], &[], None, 0, 1), Err(Refusal::NoopWithEmptyAxes)); + // The silent twin: the same call inside the subset answers. + assert!(r(&[8], &[], None, 0, 0).is_ok()); +} + +/// FAILS IF: an int32 sum that leaves int32 wraps (as a naive i32 loop would) +/// instead of being refused; a sum that fits is still answered. +#[test] +fn int32_overflow_is_refused_not_wrapped() { + let x = vec![i32::MAX, 1]; + let exact = i64::from(i32::MAX) + 1; + assert_eq!( + reduce_sum_where_greater(&x, &[2], 0, &[], None, 0, 0), + Err(Refusal::OutOfDtypeRange(exact)) + ); + assert_eq!( + x.iter().fold(0i32, |a, &b| a.wrapping_add(b)), + i32::MIN, + "what wrapping would answer" + ); + assert_eq!( + reduce_sum_where_greater(&x, &[2], 1, &[], None, 0, 0), + Ok(Output { + value: i32::MAX, + keepdims: false + }) + ); +} + +/// FAILS IF: the native op stops refusing an i64 overflow (2^32 rows of +/// i32::MAX would be needed for real data, so this pins the error type the +/// wrapper forwards). +#[test] +fn native_refusal_is_forwarded() { + assert_eq!( + format!( + "{:?}", + Refusal::Native(FoldError::Overflow(lance_graph_r2il::INT_ADD)) + ), + format!( + "Native({:?})", + FoldError::Overflow(lance_graph_r2il::INT_ADD) + ) + ); +} diff --git a/crates/math-polyfill-probe/tests/polyfill.rs b/crates/math-polyfill-probe/tests/polyfill.rs new file mode 100644 index 000000000..34ce4c412 --- /dev/null +++ b/crates/math-polyfill-probe/tests/polyfill.rs @@ -0,0 +1,274 @@ +//! D-MPF-1 gates. Every answer is compared with `oracle`, a row loop that +//! shares no code with the procedures, the dialect or mask-risc. + +use lance_graph_mask_risc::{words_for, Foreign, LaneRef, Planes}; +use lance_graph_r2il::procs::{count, select_lt}; +use lance_graph_r2il::{ + FoldDialect, FoldError, Val, INT_ADD, INT_LESS, INT_MULT, INT_SUB, INT_S_DIV, LOAD, +}; +use lance_graph_relalg::{run, Answer, Cmp, Measure, Query, Rel}; +use math_polyfill_probe::{eval, oracle, Expr, Layout}; +use ogar_loco::Dialect; + +const L: Layout = Layout { + x: 0, + y: 1, + v: 2, + valid: 0, +}; + +struct Fix { + x: Vec, + y: Vec, + v: Vec, + valid: Vec, +} + +/// `n` rows; every 7th `v` is NULL. Values are signed so a sum can cancel. +fn fix(n: usize) -> Fix { + let mut valid = vec![0u64; words_for(n)]; + for r in (0..n).filter(|r| r % 7 != 3) { + valid[r / 64] |= 1 << (r % 64); + } + Fix { + x: (0..n).map(|i| (i % 50) as i32 - 20).collect(), + y: (0..n).map(|i| ((i * 7) % 31) as i32).collect(), + v: (0..n).map(|i| ((i * 13) % 97) as i32 - 48).collect(), + valid, + } +} + +fn with_planes(f: &Fix, body: impl FnOnce(&Planes<'_>) -> R) -> R { + let lanes = [LaneRef::I32(&f.x), LaneRef::I32(&f.y), LaneRef::I32(&f.v)]; + let masks: [&[u64]; 1] = [&f.valid]; + let planes = Planes { + n_rows: f.v.len(), + masks: &masks, + lanes: &lanes, + }; + body(&planes) +} + +/// Empty, partial and full selections, and several multipliers. +const EXPRS: [Expr; 5] = [ + Expr { + lo: 100, + hi: 31, + k: 3, + }, // x ≥ 100 keeps nothing + Expr { + lo: -5, + hi: 20, + k: 3, + }, // partial + Expr { + lo: -20, + hi: 31, + k: 0, + }, // every present row + Expr { + lo: 0, + hi: 10, + k: -7, + }, // negative multiplier + Expr { + lo: -20, + hi: 31, + k: 1, + }, // every present row, k = 1 +]; + +/// FAILS IF: any procedure, the scalar arithmetic, or the NULL handling +/// diverges from the row loop, at one word, a ragged tail, and many tiles. +#[test] +fn the_polyfill_expression_equals_the_oracle() { + for n in [64, 1_000, 4_096, 65_537] { + let f = fix(n); + for e in EXPRS { + let (_, want) = oracle(&f.x, &f.y, &f.v, &f.valid, e); + let got = with_planes(&f, |p| { + let mut d = FoldDialect::new(p, &Foreign::NONE); + eval(&mut d, L, e) + }); + assert_eq!(got.ok(), want, "n = {n}, {e:?}"); + } + } +} + +/// FAILS IF: the fixture stops exercising what it claims (anti-vacuity): one +/// case must select nothing, one must select a strict subset, and NULL rows +/// must exist inside the otherwise-selected range. +#[test] +fn the_fixture_covers_empty_partial_and_null() { + let f = fix(4_096); + let sel = |e| oracle(&f.x, &f.y, &f.v, &f.valid, e).0; + let present = f + .valid + .iter() + .map(|w| u64::from(w.count_ones())) + .sum::(); + assert_eq!(sel(EXPRS[0]), 0); + let partial = sel(EXPRS[1]); + assert!( + partial * 10 > present && partial < present, + "partial = {partial} of {present}" + ); + assert_eq!(sel(EXPRS[2]), present); + assert!(present < 4_096, "no NULL rows"); +} + +/// FAILS IF: the NULL plane stops gating the sum. With every row present the +/// answer must change, so the plane is load-bearing (can-fire), and the +/// polyfill must still match the oracle on the all-present fixture. +#[test] +fn null_rows_change_the_answer() { + let mut f = fix(1_000); + let e = EXPRS[2]; + let with_nulls = oracle(&f.x, &f.y, &f.v, &f.valid, e).1; + for w in f.valid.iter_mut() { + *w = u64::MAX; + } + *f.valid.last_mut().unwrap() &= (1u64 << (1_000 % 64)) - 1; + let all_present = oracle(&f.x, &f.y, &f.v, &f.valid, e).1; + assert_ne!(with_nulls, all_present); + let got = with_planes(&f, |p| eval(&mut FoldDialect::new(p, &Foreign::NONE), L, e)); + assert_eq!(got.ok(), all_present); +} + +/// FAILS IF: the two frontends over the same substrate disagree: relalg's +/// `Query` (normalize + lower) and the procedures must count the same rows. +#[test] +fn relalg_and_the_procedures_count_the_same_rows() { + let f = fix(4_096); + for e in EXPRS { + let q = Query { + rel: Rel::And(vec![ + Rel::Plane(L.valid), + Rel::i32(L.x, Cmp::Ge, e.lo), + Rel::i32(L.y, Cmp::Lt, e.hi), + ]), + measure: Measure::Count, + }; + let rel = with_planes(&f, |p| run(&q, p, None).expect("relalg").answer); + let (want, _) = oracle(&f.x, &f.y, &f.v, &f.valid, e); + assert_eq!(rel, Answer::Count(want), "{e:?}"); + } +} + +// ── typed refusals: each is an existing FoldError, never a wrong answer ───── + +fn stack_err( + n: usize, + body: impl FnOnce(&mut FoldDialect<'_>, &mut Vec) -> Result<(), FoldError>, +) -> FoldError { + let f = fix(n); + with_planes(&f, |p| { + let mut d = FoldDialect::new(p, &Foreign::NONE); + let mut s = Vec::new(); + body(&mut d, &mut s).expect_err("must refuse") + }) +} + +fn load_i32(d: &mut FoldDialect<'_>, s: &mut Vec, lane: u16) -> Result<(), FoldError> { + s.push(Val::Scalar(i64::from(lane))); + d.call(LOAD, [1, 0, 0], s) +} + +/// FAILS IF: lane + lane arithmetic is silently accepted. The fold path has +/// no element-wise value arithmetic; that is the measured Stage 1 gap. +#[test] +fn lane_plus_lane_is_refused() { + let e = stack_err(64, |d, s| { + load_i32(d, s, L.x)?; + load_i32(d, s, L.y)?; + d.call(INT_ADD, [0; 3], s) + }); + assert_eq!(e, FoldError::LaneVersusLane(INT_ADD)); +} + +/// FAILS IF: an unsigned lane compare is coerced onto the signed one. +#[test] +fn unsigned_compare_is_refused() { + let e = stack_err(64, |d, s| { + load_i32(d, s, L.x)?; + s.push(Val::Scalar(0)); + d.call(INT_LESS, [0; 3], s) + }); + assert_eq!(e, FoldError::NoUnsignedLaneCompare(INT_LESS)); +} + +/// FAILS IF: scalar overflow wraps instead of refusing. The silent twin +/// (a product that fits) must succeed, so the guard is not always-on. +#[test] +fn scalar_overflow_is_refused_and_a_fitting_product_is_not() { + let e = stack_err(64, |d, s| { + s.push(Val::Scalar(i64::MAX / 2 + 1)); + s.push(Val::Scalar(2)); + d.call(INT_MULT, [0; 3], s) + }); + assert_eq!(e, FoldError::Overflow(INT_MULT)); + let f = fix(64); + let ok = with_planes(&f, |p| { + let mut d = FoldDialect::new(p, &Foreign::NONE); + let mut s = vec![Val::Scalar(i64::MAX / 2), Val::Scalar(2)]; + d.call(INT_MULT, [0; 3], &mut s).map(|()| s) + }); + assert_eq!(ok.unwrap(), vec![Val::Scalar(i64::MAX / 2 * 2)]); +} + +/// FAILS IF: an expression whose exact value leaves i64 is answered. The +/// oracle says it does not fit; the polyfill must refuse, not wrap. +#[test] +fn an_expression_that_leaves_i64_is_refused() { + let f = fix(1_000); + let e = Expr { + lo: -20, + hi: 31, + k: i64::MAX, + }; + assert_eq!(oracle(&f.x, &f.y, &f.v, &f.valid, e).1, None); + let got = with_planes(&f, |p| eval(&mut FoldDialect::new(p, &Foreign::NONE), L, e)); + assert_eq!(got, Err(FoldError::Overflow(INT_MULT))); +} + +/// FAILS IF: the final subtraction wraps. One selected row with `v = -2` +/// and `k = i64::MAX`: the product `k * 1` fits, `-2 - i64::MAX` does not. +/// Found by review on #1752; the oracle already said `None`. +#[test] +fn an_expression_that_overflows_at_the_subtraction_is_refused() { + let f = Fix { + x: vec![0], + y: vec![0], + v: vec![-2], + valid: vec![1], + }; + let e = Expr { + lo: 0, + hi: 1, + k: i64::MAX, + }; + assert_eq!(oracle(&f.x, &f.y, &f.v, &f.valid, e), (1, None)); + let got = with_planes(&f, |p| eval(&mut FoldDialect::new(p, &Foreign::NONE), L, e)); + assert_eq!(got, Err(FoldError::Overflow(INT_SUB))); + // Silent twin: the same subtraction with room to spare answers. + let fits = Expr { + k: i64::MAX - 2, + ..e + }; + let got = with_planes(&f, |p| { + eval(&mut FoldDialect::new(p, &Foreign::NONE), L, fits) + }); + assert_eq!(got, Ok(-2 - (i64::MAX - 2))); +} + +/// FAILS IF: division by zero answers instead of refusing. +#[test] +fn division_by_zero_is_refused() { + let e = stack_err(64, |d, s| { + select_lt(d, s, L.x, 0)?; + count(d, s)?; + s.push(Val::Scalar(0)); + d.call(INT_S_DIV, [0; 3], s) + }); + assert_eq!(e, FoldError::DivisionByZero(INT_S_DIV)); +}