Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .claude/board/entries/2026-11-03-d-global-where-p0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# D-GLOBAL-WHERE P0: direct-routed virtual SoA views across selections, plus the DuckDB parity gate

**Status:** MEASURED, lab only. No production change.
**Probe:** `.claude/probes/2026-11-03-d-global-where/` (`REPORT.md`; `probe/` with binaries `falsify` and `bench`; `duckdb/` scripts; `results/`).

**Falsifier: 21/21 checks.** The F4 (DR alignment) and F6 (presence clear) guards were each disabled and turned red.
- Production dt64 DR routing (`LocalMap`, `MapTable::{map, map_at, locate}`, `Frame`) reconciles two selections holding the same global id at different local positions.
- A join on local indices gives 32 against an oracle of 14.
- Deleting an edge leaves the identity resolvable and turns the answer false.
- The XOR consistency check detects a single flipped bit.
- Folds allocate 0 bytes.
- `where_dr3::LocalSelection` projects bit-identically for two methods. Its `SchemaChanged` on a second method is a coupling, not a correctness need; Scope refuses on its own.

**DuckDB 2.0.0 gate:** 336/336 answers agree; 8 workloads × DR64 / DR256 / multi-tile (16×DR256) × sparse / medium / dense / skewed, at threads 1 and 4.
- Warm best arm: 12/0/0 W/P/L for every workload.
- But DuckDB's fixed per-query cost (~600 µs; 703 µs at 65,536 rows against 12,106 µs at 10M) dominates most warm rows. Those are not claimed as algorithmic wins.
- Work-avoided rows: W4 3-hop on `multi_dense` is 131 µs against 1,500,724 µs; on `multi_skewed` 89 µs against 57,066 µs, where DuckDB builds 3,726,779 join rows for a 1,164 answer.
- Losses: cold on every `multi_*` dataset, and W7 per-call routing. Both come from addressing: `MapTable::locate` reads every listed tile per id. Routing once and reusing costs 5 µs against 650–970 µs.
- Plan choice by cost: `csr` wins W3 everywhere. For W4, `csr` wins on small or sparse data and `front` on medium, dense and skewed. Tile-first never won.

**OPEN:**
- Scale ≥10M rows, unmeasured on our side.
- Owner: may one selection serve several MethodRefs under one Scope (EXO-46)?
- Owner: is a global → tile index wanted beside `locate`'s ambiguity scan?
3 changes: 2 additions & 1 deletion .claude/board/entries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,12 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
opposite directions; the stranding this convention prevents shows up in
exactly one of them, never both.

564 entries, 2026-08-06 .. 2026-11-03.
565 entries, 2026-08-06 .. 2026-11-03.

| date | entry id | finding | file |
|---|---|---|---|
| 2026-11-03 | `D-XLOG-META-WIRE-P0` | | [2026-11-03-d-xlog-meta-wire-p0.md](2026-11-03-d-xlog-meta-wire-p0.md) |
| 2026-11-03 | `D-GLOBAL-WHERE` | | [2026-11-03-d-global-where-p0.md](2026-11-03-d-global-where-p0.md) |
| 2026-11-03 | `D-DR-FOLD-MAP` | | [2026-11-03-d-dr-fold-map.md](2026-11-03-d-dr-fold-map.md) |
| 2026-11-03 | `D-COGNITIVE-POLYFILL` | | [2026-11-03-d-cognitive-polyfill-p0.md](2026-11-03-d-cognitive-polyfill-p0.md) |
| 2026-11-03 | `D-BAG-FOLD` | | [2026-11-03-d-bag-fold-witnessed-existential.md](2026-11-03-d-bag-fold-witnessed-existential.md) |
Expand Down
160 changes: 160 additions & 0 deletions .claude/probes/2026-11-03-d-global-where/REPORT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# D-GLOBAL-WHERE P0: direct-routed virtual SoA views across selections

**Status:** MEASURED, lab only.
- Falsifier `falsify`: 21/21 checks. Two guards were disabled to prove they carry weight: F4 and F6 both go red.
- Benchmark against DuckDB 2.0.0: all 336 answer comparisons agree (168 of ours × threads 1 and 4).
- No production change. dt64, Quack, mask-risc and ndarray are used unchanged. No Gather, no new identity table, no witness store, no per-object allocation.

Machine: 4 vCPU Intel Xeon @ 2.10 GHz, rustc 1.99.0 release, DuckDB 2.0.0 (Python).

```sh
cd .claude/probes/2026-11-03-d-global-where/probe
cargo +1.99.0 run --release -q --bin falsify # exit 1 on any failed check
cargo +1.99.0 run --release -q --bin bench -- $OUT # writes datasets + ours.json
python3 -I ../duckdb/bench_duckdb.py $OUT --threads 1,4 --runs 201
python3 -I ../duckdb/compare.py $OUT # exit 1 on any answer mismatch
```

Results are in `results/`: `falsify.log`, `bench.log`, `ours.json`, `duckdb.json`, `compare.log` (full tables), and `explain/` (EXPLAIN ANALYZE for `dr256_medium` and `multi_skewed`). The datasets are deterministic (SplitMix64) and are regenerated by `bench`, not committed.

## 1. Routing contract: what exists

| concern | mechanism | where | class |
|---|---|---|---|
| global → local, local → global | `LocalMap::local` (counted binary search) / `LocalMap::global` (pointer-table read) | `dt64/src/dr.rs:604-616` | PRODUCTION |
| a bijection within one tile | `LocalMap::prepare` refuses `Duplicate` and `TooMany` | `dr.rs:567-587` | PRODUCTION |
| identity across tiles | `MapTable::locate(frame, tiles, g)` reads EVERY listed tile and refuses an id held twice (`Ambiguous`) | `dr.rs:950-975` | PRODUCTION |
| tile width from the survivor count | `Axis::for_survivors` (DR64 up to 64, DR256 up to 256, then ⌈n/256⌉ DR256) | `dr.rs:63-69` | PRODUCTION |
| version and frame | `MapTable::map(frame)` replays the layout the frame was sealed under; foreign frame → `Stale(ForeignFrame)`; `map_at` future → `FutureVersion` | `dr.rs:909-930` | PRODUCTION |
| frame currency | `FrameStore::check` names the first revision that moved | `dr.rs:442-462` | PRODUCTION |
| relation presence, one shared map | `xlog::Cube` (`new`, `new_tiled`, `add_edge`/`retire_edge` through a `DeltaLog`) | `xlog/src/cube.rs` | PRODUCTION crate, lab callers |
| Scope / Focus | `Scope { id }`: "Policy, checked as its own guard; never part of frame identity"; `Focus { revision }` | `dr.rs:493-505` | PRODUCTION |
| WHERE → one frame, admission | `where_dr3::project` / `LocalSelection::admit` | `dr-math-binding-probe/src/where_dr3.rs:176, 225` | LAB |
| RelationshipMap / MapPalette / ThoughtMapPaletteBinding | no Rust type with these names (`lance-graph/src/config.rs::RelationshipMapping` is the unrelated Cypher config) | plans D-RMAP-1A | PROPOSED |

**Answer to the operator's model:** DR already owns exact, bidirectional, versioned routing, and the cross-selection join needs nothing beyond it.
- Two selections that hold the same global Y at different local positions are reconciled through the global id: `global()` on one side, then `local()` or `locate` on the other.
- Comparing local indices directly is wrong. F4 measures it: 32 by local coincidence against an oracle of 14.

## 2. Falsifier (Case A and Case B), `falsify`

Fixture:
- A universe of 65,536 ids.
- A relation domain D of 512 ids over two DR256 tiles (the R frame).
- Selection tiles 10 (A, descending order) and 11 (B, ascending order).
- A presence plane over R coordinates.

| # | check | result |
|---|---|---|
| F0 | Quack `Keep` (`a<50`) = lane oracle at all 65,536 ids | PASS (\|A\|=3296, \|B\|=19584) |
| F1/F2 | G→L→G and L→G→L for 714 ids/locals of tiles 0, 1, 10, 11 | PASS |
| F3 | the same global y has local 1 under A and 156 under B; 14 ids shared | PASS |
| F4 | local-coincidence join = 32 (wrong); DR-aligned = 14 = oracle | PASS; **disable (join by local index) → FAIL** |
| F5/F18 | Case B: x ∈ A in R tile 0, y ∈ B in R tile 1; `row[x] & selB_R ≠ 0` true = oracle; Q (x ∈ A with a B-neighbour) = 23 = oracle | PASS |
| F6 | clear that edge's presence bit: `locate(y)` still Ok, answer false, Q 23 → 22 = oracle | PASS; **disable (bit not cleared) → FAIL** |
| F7 | clear an unrelated edge: Q unchanged | PASS |
| F8/F9 | forward plane XOR transpose(reverse) = 0; one flipped bit → exactly one differing position, at that edge | PASS |
| F10 | historical frame: after a tile-10 replacement, f1 reads the old locals and f2 the new (snapshot replay); coordinates revision moved | PASS |
| F11 | foreign frame → `Stale(ForeignFrame)`; future version → `FutureVersion`; `locate` of the shared id in [10, 11] → `Ambiguous`; one tile → Ok | PASS |
| F12 | foreign-class row axis → `AxisMismatch` | PASS |
| F13a | M1's selection admitted against M2's register (same axis) → `SchemaChanged` | PASS |
| F13b | the projections for M1 and for M2 are bit-identical (truth and slots) | PASS |
| F13c | same method, another Scope → `ScopeChanged` | PASS |
| F14/F15 | the folds of F5 and F7 allocate 0 bytes in 0 calls; no tuple vector | PASS |
| F16 | probe types: no identity store besides MapTable/LocalMap (manual) | PASS (manual) |
| F17 | Boolean one-hop distinct 156 ≠ Bag 2-path count 1095, both = oracle; `bind_dr64(Bag)` → `BagNeedsMultiplicity` | PASS |

**F13 settles the operator's coupling question.**
- `LocalSelection` couples two concerns: its projected bits depend only on (map, WHERE mask, frame) (13b), yet `admit` also compares the full `MethodSignature` (13a).
- The authorization guard is separate, and the method check is not needed to enforce it: Scope refuses on its own (13c).
- So `SchemaChanged` on a second method over the same axis is an unnecessary coupling for correctness. Whether two MethodRefs may share one selection stays an authorization decision, which Scope already expresses.
- **Caveat:** this is the lab `LocalSelection`; production has no selection type at all (§1).

## 3. DuckDB parity gate

**Gate:** parity or better on clearly identified workloads with exact semantic equivalence; every losing workload is recorded with its cause.
- Answers: 336/336 agree.
- DuckDB runs with its normal optimizer at threads 1 and 4, PREPARE/EXECUTE, 201 warm runs. The Python call floor (median 179 µs, `PREPARE f AS SELECT 1`) is subtracted.
- Ours is single-threaded; "warm" includes the WHERE evaluation and the fold.

### 3.1 What the warm numbers do and do not show

**DuckDB's fixed cost per query dominates every warm comparison at these sizes.** Measured on the same connection (wall, Python included):
- `SELECT 1`: 93 µs
- `count(*) FROM nodes` (65,536 rows): 297 µs
- `count(*) … WHERE a<50` over 65,536 rows: 703 µs
- the same over 10,000,000 rows: 12,106 µs

That is roughly 600 µs fixed per query and about 1.2 ns per filtered row. Our W1 (Quack `Count` over the same 65,536-row lane) is 5.7–10.2 µs, about 0.15 ns/row.

So most warm "wins" (W1, W2, W5a, W7, W7i, W8 on every dataset; most workloads on DR64/DR256) are DuckDB's per-query overhead against an in-process fold. **They are not evidence of algorithmic superiority, and are not claimed as such.** The domain here is at most 4,096 nodes; nothing was measured at the scale where DuckDB's per-row throughput dominates. At 10M rows, our side was not measured (PROBE).

### 3.2 Where the difference is work avoided, not overhead

Rows where DuckDB's own engine time is far above its fixed cost. Warm, µs, DuckDB minus floor:

| dataset | workload | ours (best arm) | DuckDB t=1 | DuckDB t=4 | DuckDB intermediate |
|---|---|---|---|---|---|
| multi_dense (503,921 edges) | W4 3-hop distinct | 130.7 (`front`) | 1,500,724 | 418,289 | — |
| multi_skewed (32,648 edges) | W4 3-hop distinct | 89.2 (`front`) | 57,066 | 56,569 | 3,726,779 join rows for a 1,164 answer (`explain/multi_skewed_W4.txt`) |
| multi_dense | W3 2-hop EXISTS | 7.7 (`csr`) | 29,097 | 17,989 | — |
| multi_dense | W6 bag 2-paths (3,382,676) | 119.3 | 22,726 | 11,379 | — |
| multi_skewed | W3 | 7.6 (`csr`) | 6,767 | 6,391 | — |

Here the frontier masks never materialize the join, while DuckDB builds and aggregates the intermediate. This is the intended win: work the query never needed.

### 3.3 Losing workloads and their cause (all recorded)

| workload | where | ours | DuckDB | cause |
|---|---|---|---|---|
| **cold** (prep + first query) | all 4 `multi_*` datasets, every workload except W1 (counts 8/0/4 and 9/0/3) | 8,746 – 519,891 µs | 1,852 – 35,491 µs (PREPARE + first EXECUTE; load is separate, 40,000 – 117,000 µs) | **addressing**: the presence plane is built with `MapTable::locate` per edge endpoint. `locate` reads every listed tile (16 here) to refuse duplicates, so it costs O(tiles) per id. On `multi_dense` that is about 1M endpoints × 16 binary searches. Including DuckDB's load, ours is lower on sparse/medium/skewed and higher on dense (516 ms vs 98 + 7 ms). |
| W7 `route_each` | `multi_*` | 650 – 970 µs (FAILED-GATE at ≥1000 µs on `multi_skewed`) | 1,488 – 3,328 µs | **addressing**: routing selB (~1,181 ids) costs one `locate` × 16 tiles per id. Routing once and reusing it (`reuse` arm) takes 4.5 – 6.4 µs; that is the arm counted. |
| update (W8 delete) | `multi_dense` (5,039 edges) | 4,822 µs FAILED-GATE (not a loss to DuckDB) | DELETE 9,685 µs (t=1) / 5,787 µs (t=4) | **addressing**: the same per-edge `locate`; over the workspace's 1 ms gate |
| W4 `csr` arm | `multi_dense` | 1,127 µs (FAILED-GATE) | — | **plan choice**: indexed navigation loses to frontier masks on dense data; `front` (131 µs) is the counted arm |

No warm best-arm workload lost to DuckDB (W/P/L 12/0/0 for every workload at both thread counts). Given §3.1, this does **not** establish general superiority.

### 3.4 Plan choice by measured cost

- **W3 (EXISTS 2-hop):** `csr` (indexed DFS with early exit) is fastest on every dataset (0.3 – 10.6 µs). `back` (selection-first backward mask) and `tile` (per-x 2-hop row) are 2 – 60× slower.
- **W4 (3-hop distinct):** `csr` wins on DR64/DR256 and `multi_sparse`. Selection-first frontier masks (`front`) win on `multi_medium`/`multi_dense`/`multi_skewed` (110/131/89 µs against csr 2–14× slower).
- So the operator's three plans each win somewhere, and the choice must be made by measured cost.
- The crossover is fan-out × frontier width.
- Tile-first (`tile`) never won here.

### 3.5 Allocation and memory

- `alloc_query` is 0 bytes for every warm arm (`ours.json`).
- The presence plane costs n²/8 bytes: 2 MiB for 4,096 nodes, built at cold time.
- Peak bytes per dataset are in `ours.json`; DuckDB `duckdb_memory()` totals are in `duckdb.json` `mem_bytes`.

## 4. Verdict

**ALREADY WORKS**
- Production DR provides exact, bidirectional, versioned, multi-tile routing: `LocalMap`, `MapTable::{map, map_at, locate}`, `Frame`/`FrameStore`, `Axis::for_survivors`.
- A cross-selection reference resolves through the global id with no copied object, no new relation table and no new identity store. Measured: F1–F5, F10, F11, F18.
- Presence change through the relation plane works: F6, F7.
- XOR consistency checking works: F8, F9.
- Quack WHERE → Keep is exact (F0).
- Folds allocate nothing (F14, F15).

**WIRE NOW** (lab-proven, small)
- Route a selection into the relation frame ONCE per frame and reuse it.
- Every multi-tile loss above is per-id `locate`, never the fold. Routing once is what `reuse` (5 µs vs 650–970 µs) and the Cube's "each id looked up once at construction" already do.

**OWNER DECISION**
- (a) The coupling in `LocalSelection::admit` (`where_dr3.rs:244`): F13 shows the projection is method-independent and Scope already refuses separately. May one admitted selection serve several MethodRefs under one Scope? (EXO-46)
- (b) Whether a global → tile index (one lookup instead of reading every listed tile) is wanted beside `locate`'s ambiguity scan.
- `locate`'s O(tiles) read is deliberate: it refuses an id held twice.
- The cost it puts on multi-tile cold paths is measured above.
- (c) No production type yet holds a relation presence plane whose two axes come from one frame, outside `Cube` internals.

**PROBE**
- Throughput at a scale where DuckDB's per-row cost dominates (≥10M rows, wider domains). This is the gate that remains unanswered.
- DR256 multi-tile plane build via `Cube::new_tiled` instead of the probe's own `Plane`.

**PARK**
- General recursion (EXO-73), non-Boolean Bag expansion beyond the W6 degree product (EXO-108), physical disk prefetch, unrelated runtime work.

**SKIP**
- Duplicate witnesses, per-object allocation, a second routing store, mask transfers across programs, new Gather operators, complete executor migration.
Loading
Loading