From 0a406a5f7c8d6e6ebca8f82dcd593d2919b29332 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 08:06:37 +0000 Subject: [PATCH 1/2] Probe P-VIEW-1: a saved-view page costs 62-80 scans for an I32 key 25-record pages at 1M and 4M rows. An I32 key runs 67 Count programs and takes about twice as long as sorting every kept record at the edge; a code key costs about 1.7 scans. The rank-select / ORDER BY ask goes upstream (ISS-ORDER-BY-RANK-SELECT). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z --- .claude/board/ISSUES.md | 15 + .claude/board/LATEST_STATE.md | 10 +- .claude/board/STATUS_BOARD.md | 2 +- .../hubspo-query/examples/probe_view_page.rs | 308 ++++++++++++++++++ 4 files changed, 332 insertions(+), 3 deletions(-) create mode 100644 crates/hubspo-query/examples/probe_view_page.rs diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index f7458e9..03bd88b 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -3,6 +3,21 @@ > Open blockers and cross-repo follow-ups. Append new entries at the top; > change only the `Status` line of an existing entry. +## ISS-ORDER-BY-RANK-SELECT — a page sorted by an `I32` key costs more than sorting everything + +- **Status:** OPEN 2026-10-21. Measured by P-VIEW-1; the ask is posted to lance-graph on EXO-50 (operator's instruction: probe first, then post with numbers). HubSPO-rs builds nothing in place of it. +- **Where:** `hubspo-query::view::page`, the `I32` arm (`value_at` bisection). +- **What:** without a rank-select or `ORDER BY` terminal, the page's first and last value are found by bisection: 67 `Count` programs per page whatever the offset. Measured (`cargo run --release -p hubspo-query --example probe_view_page`, 25-record pages, median of 5): + + | rows | one threshold scan | `I32` page | full edge sort | code-key page | + |---|---|---|---|---| + | 1M | 1.16 ms | 80–93 ms | 40–46 ms | 1.9–2.2 ms | + | 4M | 5.71 ms | 355–405 ms | 196–230 ms | 8.9–9.3 ms | + + An `I32` page is ~62–80 scans and about 2× a full sort at the edge. A code-key page is ~1.6–1.9 scans but materialises the whole boundary code (~25 % of rows with 4 codes). Costs scale linearly with rows. +- **Ask:** one program that returns the value at a given rank of an `I32` lane over a filter (rank select), or a top-k / `ORDER BY` terminal returning the ordinals of ranks `[offset, offset + k)` with ties by ordinal. Either replaces the 66 bisection programs with one scan. + + ## ISS-PRESET-VERBS-PROVISIONAL — the preset roles' verbs are written from seat names, not from HubSpot - **Status:** DECIDED 2026-10-21 (operator, EXO-50 message 006): AMENDED, not approved. The six role names stay. The verb table stays only as a tagged compatibility/simulation fixture, never as tenant runtime access. EffectiveAccess = seat/subscription ceiling ∧ tenant permission grants ∧ owner/team scope ∧ per-property outer+ordinal authorization ∧ action-specific gates. Super Admin is an explicit status, not a bypass of tenant isolation, schema/policy validation or seat gates. Admin is not blanket FULL/DELETE. View-only has no implicit read of sensitive fields. ACT (email send, enrolment, invoice, bulk), DELETE and EXPORT each need an explicit grant and are never inherited from WRITE/CREATE. A missing, unknown, stale or unread tenant grant fails closed and never falls back to the fixture. The presets are not reported as HubSpot-verified until a sandbox oracle exists (ISS-ORACLE-PORTAL). Implemented in the D-IAM-3 code PR (2026-10-21): the runtime bind grants nothing, the table is `PresetFixture`. diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 99b2e1d..5d27761 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -217,8 +217,14 @@ `before` count ignored, nulls dropped, bisection off by one, code order not reversed, band missing the view's filter, sort left out of the gate, hidden column kept. -- Open: a page-cost probe at 1M+ rows (the `ORDER BY` question waits on - it), multi-key sorts, view sharing (owner/team/everyone) and persistence. +- P-VIEW-1 (`examples/probe_view_page.rs`, 2026-10-21): 25-record pages + at 1M and 4M rows. `I32` key: 67 programs, 80–93 ms at 1M and 355–405 ms + at 4M, 62–80× one threshold scan and ~2× sorting every kept record at the + edge. Code key: 3 programs, ~1.7 scans, but the whole boundary code is + materialised. Linear in rows. The probe failed for `I32`; the rank-select + / `ORDER BY` ask is on EXO-50 (`ISS-ORDER-BY-RANK-SELECT`). +- Open: that ask, multi-key sorts, view sharing (owner/team/everyone) and + persistence. ## Lists (D-QRY-2) diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 9d87bf0..996016a 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -49,7 +49,7 @@ |---|---|---|---|---| | D-QRY-1 | Filter compiler: HubSpot filter groups (AND within group, OR across groups; operators EQ/NEQ/LT/…/HAS_PROPERTY/CONTAINS_TOKEN/IN/BETWEEN) → Quack program | L | Shipped | 2026-10-14: lance-graph #1546 `lance-graph-relalg` (D-RA-1) normalizes a filter before it runs and answers a contradictory one as `Empty` without a program; D-CMR-1 moves that set algebra into the contract. Adopt it there, not from the prototype crate; `hubspo-store::catalog` + `hubspo-query::filter`; 600 random filter sets equal a plain-loop oracle on mask-risc; #19 | | D-QRY-2 | Lists: active (incremental on version delta) + static (stored mask); membership history | L | Shipped | `hubspo-query::list` (refresh, static add/remove/members, added/removed); tests vs full recomputation. P-LIST-1 2.6 ms at 10M; P-SPARSE-1 1.93 ms, 257 KiB flat at 10M. Storage of list masks lands with D-STO-1 | -| D-QRY-3 | Saved views + board view (report pivot over pipeline × stage) | M | In progress | board half shipped (#21): `hubspo-query::board` (count + amount per stage as two `GroupReduce` programs); saved-view half shipped 2026-10-21: `hubspo-query::view` (`SavedView`, `page` with edge sort: storage order, `Count`-threshold bisection for I32, `GroupReduce Count` for codes; ties by ordinal, nulls last), oracle over every page, `hubspo-compat::views::open_view` through the D-IAM-3 gate; 8 disable runs red. Open: page-cost probe at 1M+ rows (ORDER BY asked upstream only if it fails), multi-key sort, sharing, persistence | +| D-QRY-3 | Saved views + board view (report pivot over pipeline × stage) | M | In progress | board half shipped (#21): `hubspo-query::board` (count + amount per stage as two `GroupReduce` programs); saved-view half shipped 2026-10-21: `hubspo-query::view` (`SavedView`, `page` with edge sort: storage order, `Count`-threshold bisection for I32, `GroupReduce Count` for codes; ties by ordinal, nulls last), oracle over every page, `hubspo-compat::views::open_view` through the D-IAM-3 gate; 8 disable runs red. P-VIEW-1 2026-10-21: the probe failed for `I32` keys (a page is 62–80 scans, ~2× a full edge sort; 82 ms at 1M, ~380 ms at 4M) and passed for code keys (~1.7 scans); rank-select/ORDER BY asked upstream on EXO-50 (`ISS-ORDER-BY-RANK-SELECT`). Open: that ask, multi-key sort, sharing, persistence | | D-QRY-4 | Quack lowering of callcenter's `ParsedQuery` (the upstream DataFusion `parsed_query_to_plan` is an unimplemented stub) | M | Queued | | | D-QRY-5 | Association-traversing filters ("contacts whose company has an open deal"): Quack `Semijoin` over edge lanes for one hop; multi-hop needs cypher-quack shapes that do not exist yet → upstream request first | L | Queued | 2026-10-20: lance-graph #1632/#1633 (D-SRAM-2 lab spike, `crepe-plan-probe`) measured a semijoin reduction before preparation and an addressed semijoin over attested lanes with masked Lance 13 reads; lab only, not authorization; 2026-10-14: the hop is D-CMR-5, preimage `f⁻¹(S)` under a proven mapping (lance-graph `d-cmr-concept-chain-normalization-v1`), not chained key reads (withdrawn in #1524 / ndarray #359). One hop is ready on the existing Quack `Semijoin`; multi-hop waits on D-CMR-5 rather than on new cypher-quack shapes | | D-QRY-6 | Reports: single-object, cross-object, funnel, time-series, attribution skeleton on `lance-graph-report` | XL | Queued | 2026-10-21 ruling: also owns forecast types (`crm-forecast-types`, 2 functions) and reads D-FLOW-2's custom behavioural events. | diff --git a/crates/hubspo-query/examples/probe_view_page.rs b/crates/hubspo-query/examples/probe_view_page.rs new file mode 100644 index 0000000..047f059 --- /dev/null +++ b/crates/hubspo-query/examples/probe_view_page.rs @@ -0,0 +1,308 @@ +//! P-VIEW-1: what one saved-view page costs without `ORDER BY`. +//! +//! For each sort key and offset, one 25-record page through +//! `hubspo_query::view::page`, against two references on the same data: +//! +//! - **full sort:** every kept record materialised, sorted by (value, +//! ordinal), sliced: what a page costs if the edge sorts everything; +//! - **one scan:** a single threshold `Agg::Count` over the key lane: a floor +//! for any in-engine `ORDER BY`, which has to read the key lane at least once. +//! +//! Reported per case: programs run, rows materialised, median wall time of +//! `RUNS` repetitions. Keys: `score` (i32, near-unique), `amount` (i32 in +//! -20..20, 20 % null: heavy ties), `stage` (code, 4 values: a page inside +//! one code materialises that code). +//! +//! `cargo run --release -p hubspo-query --example probe_view_page [rows]` + +use std::time::{Duration, Instant}; + +use hubspo_query::view::{Direction, Runner, SavedView, Sort, page}; +use hubspo_store::catalog::{Catalog, Codebook, FieldSpec}; +use lance_graph_mask_risc::{ + Foreign, LaneRef, Out, Planes, Scratch, Value, execute_into, materialize_rows, words_for, +}; +use lance_graph_quack::bind::{FieldKind, create_table}; +use lance_graph_quack::{Agg, Col, Mask, Query, lower}; + +const LIMIT: usize = 25; +const RUNS: usize = 5; +const STAGES: &[&str] = &["lead", "opportunity", "customer", "evangelist"]; + +struct Rng(u64); +impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } +} + +struct Data { + n: usize, + live: Vec, + amount_valid: Vec, + stage: Vec, + amount: Vec, + score: Vec, + programs: usize, + materialised: usize, +} + +impl Data { + fn new(n: usize) -> Self { + let mut r = Rng(0x0D3E_5EED); + let mut live = vec![0u64; words_for(n)]; + let mut amount_valid = vec![0u64; words_for(n)]; + let (mut stage, mut amount, mut score) = (Vec::new(), Vec::new(), Vec::new()); + for i in 0..n { + let x = r.next(); + if x % 100 >= 5 { + live[i / 64] |= 1 << (i % 64); + } + if (x >> 8) % 100 >= 20 { + amount_valid[i / 64] |= 1 << (i % 64); + } + stage.push(((x >> 16) % 4) as u32); + amount.push(((x >> 24) % 41) as i32 - 20); + score.push(r.next() as i32); + } + Self { + n, + live, + amount_valid, + stage, + amount, + score, + programs: 0, + materialised: 0, + } + } + + fn run(&mut self, q: &Query, out: Out<'_>) -> Value { + self.programs += 1; + let masks: [&[u64]; 2] = [&self.live, &self.amount_valid]; + let lanes = [ + LaneRef::U32(&self.stage), + LaneRef::I32(&self.amount), + LaneRef::I32(&self.score), + ]; + let planes = Planes { + n_rows: self.n, + masks: &masks, + lanes: &lanes, + }; + let program = lower(q).expect("lower"); + let mut scratch = Scratch::for_program(&program, self.n).expect("scratch"); + execute_into(&program, &planes, &Foreign::NONE, &mut scratch, out).expect("execute") + } +} + +impl Runner for Data { + type Error = (); + fn count(&mut self, q: &Query) -> Result { + match self.run(q, Out::None) { + Value::Count(n) => Ok(n as u64), + v => panic!("{v:?}"), + } + } + fn rows(&mut self, q: &Query) -> Result, ()> { + let mut kept = vec![0u64; words_for(self.n)]; + self.run(q, Out::Mask(&mut kept)); + let rows = materialize_rows(&kept, self.n); + self.materialised += rows.len(); + Ok(rows) + } + fn groups(&mut self, q: &Query, groups: u32) -> Result, ()> { + let mut sink = vec![0i64; groups as usize]; + self.run(q, Out::I64(&mut sink)); + Ok(sink) + } + fn code_domain(&self, _: Col) -> u32 { + STAGES.len() as u32 + } + fn i32_at(&self, col: Col, row: usize) -> i32 { + match col { + Col(1) => self.amount[row], + _ => self.score[row], + } + } +} + +fn catalog() -> Catalog { + let mut c = Catalog::new(); + let t = create_table("contacts") + .width(8) + .register(&mut c) + .expect("register") + .id; + let fields = [ + ("lifecyclestage", Col(0), FieldKind::Code, None), + ("amount", Col(1), FieldKind::I32, Some(Mask(1))), + ("score", Col(2), FieldKind::I32, None), + ]; + for (name, col, kind, validity) in fields { + c.declare( + t, + FieldSpec { + name: name.into(), + col, + kind, + validity, + codebook: (kind == FieldKind::Code) + .then(|| Codebook::new(STAGES.iter().copied()).expect("codebook")), + }, + ) + .expect("declare"); + } + c +} + +fn median(mut v: Vec) -> Duration { + v.sort_unstable(); + v[v.len() / 2] +} + +fn ms(d: Duration) -> f64 { + d.as_secs_f64() * 1e3 +} + +/// Every kept record, sorted at the edge: the reference a full sort costs. +fn full_sort(d: &mut Data, key: Col, offset: usize) -> (Vec, usize) { + let live = Query { + filter: lance_graph_quack::Filter::plane(Mask(0)), + agg: Agg::Rows, + }; + let rows = d.rows(&live).expect("rows"); + let n = rows.len(); + let mut keyed: Vec<(i32, usize)> = rows + .into_iter() + .map(|r| { + let v = match key { + Col(0) => d.stage[r] as i32, + Col(1) => d.amount[r], + _ => d.score[r], + }; + (v, r) + }) + .collect(); + keyed.sort_unstable(); + ( + keyed + .into_iter() + .skip(offset) + .take(LIMIT) + .map(|(_, r)| r) + .collect(), + n, + ) +} + +fn main() { + let n: usize = std::env::args() + .nth(1) + .map(|a| a.parse().expect("rows")) + .unwrap_or(1_000_000); + let cat = catalog(); + let mut d = Data::new(n); + let live_n = d + .count(&Query { + filter: lance_graph_quack::Filter::plane(Mask(0)), + agg: Agg::Count, + }) + .unwrap() as usize; + + // One scan of the key lane: a threshold Count, the floor any in-engine + // ORDER BY pays (a bare plane Count is a popcount and reads no lane). + let scan = median( + (0..RUNS) + .map(|_| { + let t = Instant::now(); + d.count(&Query { + filter: lance_graph_quack::Filter::and([ + lance_graph_quack::Filter::plane(Mask(0)), + lance_graph_quack::Filter::cmp(Col(2), lance_graph_quack::Cmp::LeI32(0)), + ]), + agg: Agg::Count, + }) + .unwrap(); + t.elapsed() + }) + .collect(), + ); + println!("rows {n}, live {live_n}, page {LIMIT}, median of {RUNS}"); + println!( + "one scan (threshold Count on the key lane): {:.2} ms", + ms(scan) + ); + println!( + "{:<16} {:>9} {:>9} {:>12} {:>10} {:>12} {:>10}", + "key", "offset", "programs", "materialised", "page ms", "full-sort ms", "page/scan" + ); + + let keys = [ + ("score", Col(2), Direction::Ascending), + ("score", Col(2), Direction::Descending), + ("amount", Col(1), Direction::Ascending), + ("lifecyclestage", Col(0), Direction::Ascending), + ]; + for (name, col, direction) in keys { + let view = SavedView { + object: "contacts".into(), + sort: Some(Sort { + property: name.into(), + direction, + }), + ..SavedView::default() + }; + for offset in [0usize, live_n / 2, live_n - LIMIT] { + let mut times = Vec::new(); + let (mut programs, mut materialised) = (0, 0); + let mut got = Vec::new(); + for _ in 0..RUNS { + d.programs = 0; + d.materialised = 0; + let t = Instant::now(); + got = page(&view, &cat, offset, LIMIT, &mut d).expect("page"); + times.push(t.elapsed()); + programs = d.programs; + materialised = d.materialised; + } + assert_eq!(got.len(), LIMIT, "{name} {offset}"); + // The full-sort reference, and a check against it where the + // order agrees (ascending, no nulls). + let full = median( + (0..RUNS) + .map(|_| { + let t = Instant::now(); + let (want, _) = full_sort(&mut d, col, offset); + if direction == Direction::Ascending && name != "amount" { + assert_eq!(got, want, "{name} {offset}"); + } + t.elapsed() + }) + .collect(), + ); + let p = median(times); + println!( + "{:<16} {:>9} {:>9} {:>12} {:>10.2} {:>12.2} {:>10.1}", + format!( + "{name} {}", + if direction == Direction::Ascending { + "asc" + } else { + "desc" + } + ), + offset, + programs, + materialised, + ms(p), + ms(full), + p.as_secs_f64() / scan.as_secs_f64() + ); + } + } +} From 959b23b71ba3678007129918348e0febf8369dad Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Oct 2026 08:11:59 +0000 Subject: [PATCH 2/2] P-VIEW-1: tail pages over the key's valued records; null tail reported apart Codex on #46: the nullable amount key's tail offset fell in its null tail, which runs 2 programs and no bisection, so '67 programs whatever the offset' did not hold for every printed row. Offsets now come from the key's valued count, the null-tail page gets its own labelled row, and the board quotes ranges over three runs. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01PPcakjk7ioQsnasmNDtc5z --- .claude/board/ISSUES.md | 12 ++++---- .claude/board/LATEST_STATE.md | 10 ++++--- .claude/board/STATUS_BOARD.md | 2 +- .../hubspo-query/examples/probe_view_page.rs | 30 +++++++++++++++++-- 4 files changed, 40 insertions(+), 14 deletions(-) diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index 03bd88b..d8cb274 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -7,14 +7,14 @@ - **Status:** OPEN 2026-10-21. Measured by P-VIEW-1; the ask is posted to lance-graph on EXO-50 (operator's instruction: probe first, then post with numbers). HubSPO-rs builds nothing in place of it. - **Where:** `hubspo-query::view::page`, the `I32` arm (`value_at` bisection). -- **What:** without a rank-select or `ORDER BY` terminal, the page's first and last value are found by bisection: 67 `Count` programs per page whatever the offset. Measured (`cargo run --release -p hubspo-query --example probe_view_page`, 25-record pages, median of 5): +- **What:** without a rank-select or `ORDER BY` terminal, the page's first and last value are found by bisection: 67 `Count` programs for every page that has valued records, at any offset. A page wholly inside a nullable key's null tail runs 2 programs and no bisection (Codex on #46: the first run's `amount` tail page was such a page). Measured (`cargo run --release -p hubspo-query --example probe_view_page`, 25-record pages, median of 5 per run, ranges over three runs on a shared 4-core container): - | rows | one threshold scan | `I32` page | full edge sort | code-key page | - |---|---|---|---|---| - | 1M | 1.16 ms | 80–93 ms | 40–46 ms | 1.9–2.2 ms | - | 4M | 5.71 ms | 355–405 ms | 196–230 ms | 8.9–9.3 ms | + | rows | one threshold scan | `I32` page (valued) | full edge sort | code-key page | null-tail page | + |---|---|---|---|---|---| + | 1M | 1.16–2.31 ms | 79–142 ms | 38–66 ms | 2.0–3.2 ms | 0.36–0.54 ms | + | 4M | 5.34–5.85 ms | 352–421 ms | 196–242 ms | 8.9–13.9 ms | 1.5–2.3 ms | - An `I32` page is ~62–80 scans and about 2× a full sort at the edge. A code-key page is ~1.6–1.9 scans but materialises the whole boundary code (~25 % of rows with 4 codes). Costs scale linearly with rows. + A valued `I32` page is 60–100 scans (same-run ratio) and 1.6–2.8× a full sort at the edge. A code-key page is ~1–2.7 scans but materialises the whole boundary code (~25 % of rows with 4 codes). Costs scale linearly with rows. - **Ask:** one program that returns the value at a given rank of an `I32` lane over a filter (rank select), or a top-k / `ORDER BY` terminal returning the ordinals of ranks `[offset, offset + k)` with ties by ordinal. Either replaces the 66 bisection programs with one scan. diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 5d27761..0659a9e 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -218,10 +218,12 @@ not reversed, band missing the view's filter, sort left out of the gate, hidden column kept. - P-VIEW-1 (`examples/probe_view_page.rs`, 2026-10-21): 25-record pages - at 1M and 4M rows. `I32` key: 67 programs, 80–93 ms at 1M and 355–405 ms - at 4M, 62–80× one threshold scan and ~2× sorting every kept record at the - edge. Code key: 3 programs, ~1.7 scans, but the whole boundary code is - materialised. Linear in rows. The probe failed for `I32`; the rank-select + at 1M and 4M rows, ranges over three runs. `I32` key: 67 programs on every + page with valued records, 79–142 ms at 1M and 352–421 ms at 4M, 60–100× + one threshold scan and 1.6–2.8× sorting every kept record at the edge. A + page wholly in a nullable key's null tail: 2 programs. Code key: 3 + programs, ~1–2.7 scans, but the whole boundary code is materialised. + Linear in rows. The probe failed for `I32`; the rank-select / `ORDER BY` ask is on EXO-50 (`ISS-ORDER-BY-RANK-SELECT`). - Open: that ask, multi-key sorts, view sharing (owner/team/everyone) and persistence. diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 996016a..7336728 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -49,7 +49,7 @@ |---|---|---|---|---| | D-QRY-1 | Filter compiler: HubSpot filter groups (AND within group, OR across groups; operators EQ/NEQ/LT/…/HAS_PROPERTY/CONTAINS_TOKEN/IN/BETWEEN) → Quack program | L | Shipped | 2026-10-14: lance-graph #1546 `lance-graph-relalg` (D-RA-1) normalizes a filter before it runs and answers a contradictory one as `Empty` without a program; D-CMR-1 moves that set algebra into the contract. Adopt it there, not from the prototype crate; `hubspo-store::catalog` + `hubspo-query::filter`; 600 random filter sets equal a plain-loop oracle on mask-risc; #19 | | D-QRY-2 | Lists: active (incremental on version delta) + static (stored mask); membership history | L | Shipped | `hubspo-query::list` (refresh, static add/remove/members, added/removed); tests vs full recomputation. P-LIST-1 2.6 ms at 10M; P-SPARSE-1 1.93 ms, 257 KiB flat at 10M. Storage of list masks lands with D-STO-1 | -| D-QRY-3 | Saved views + board view (report pivot over pipeline × stage) | M | In progress | board half shipped (#21): `hubspo-query::board` (count + amount per stage as two `GroupReduce` programs); saved-view half shipped 2026-10-21: `hubspo-query::view` (`SavedView`, `page` with edge sort: storage order, `Count`-threshold bisection for I32, `GroupReduce Count` for codes; ties by ordinal, nulls last), oracle over every page, `hubspo-compat::views::open_view` through the D-IAM-3 gate; 8 disable runs red. P-VIEW-1 2026-10-21: the probe failed for `I32` keys (a page is 62–80 scans, ~2× a full edge sort; 82 ms at 1M, ~380 ms at 4M) and passed for code keys (~1.7 scans); rank-select/ORDER BY asked upstream on EXO-50 (`ISS-ORDER-BY-RANK-SELECT`). Open: that ask, multi-key sort, sharing, persistence | +| D-QRY-3 | Saved views + board view (report pivot over pipeline × stage) | M | In progress | board half shipped (#21): `hubspo-query::board` (count + amount per stage as two `GroupReduce` programs); saved-view half shipped 2026-10-21: `hubspo-query::view` (`SavedView`, `page` with edge sort: storage order, `Count`-threshold bisection for I32, `GroupReduce Count` for codes; ties by ordinal, nulls last), oracle over every page, `hubspo-compat::views::open_view` through the D-IAM-3 gate; 8 disable runs red. P-VIEW-1 2026-10-21: the probe failed for `I32` keys (a valued page is 67 programs, 60–100 scans, 1.6–2.8× a full edge sort; 79–142 ms at 1M, 352–421 ms at 4M) and passed for code keys (~1–2.7 scans); a null-tail page is 2 programs; rank-select/ORDER BY asked upstream on EXO-50 (`ISS-ORDER-BY-RANK-SELECT`). Open: that ask, multi-key sort, sharing, persistence | | D-QRY-4 | Quack lowering of callcenter's `ParsedQuery` (the upstream DataFusion `parsed_query_to_plan` is an unimplemented stub) | M | Queued | | | D-QRY-5 | Association-traversing filters ("contacts whose company has an open deal"): Quack `Semijoin` over edge lanes for one hop; multi-hop needs cypher-quack shapes that do not exist yet → upstream request first | L | Queued | 2026-10-20: lance-graph #1632/#1633 (D-SRAM-2 lab spike, `crepe-plan-probe`) measured a semijoin reduction before preparation and an addressed semijoin over attested lanes with masked Lance 13 reads; lab only, not authorization; 2026-10-14: the hop is D-CMR-5, preimage `f⁻¹(S)` under a proven mapping (lance-graph `d-cmr-concept-chain-normalization-v1`), not chained key reads (withdrawn in #1524 / ndarray #359). One hop is ready on the existing Quack `Semijoin`; multi-hop waits on D-CMR-5 rather than on new cypher-quack shapes | | D-QRY-6 | Reports: single-object, cross-object, funnel, time-series, attribution skeleton on `lance-graph-report` | XL | Queued | 2026-10-21 ruling: also owns forecast types (`crm-forecast-types`, 2 functions) and reads D-FLOW-2's custom behavioural events. | diff --git a/crates/hubspo-query/examples/probe_view_page.rs b/crates/hubspo-query/examples/probe_view_page.rs index 047f059..d266a6a 100644 --- a/crates/hubspo-query/examples/probe_view_page.rs +++ b/crates/hubspo-query/examples/probe_view_page.rs @@ -257,7 +257,26 @@ fn main() { }), ..SavedView::default() }; - for offset in [0usize, live_n / 2, live_n - LIMIT] { + // Offsets over the key's VALUED records, so the tail page still sorts; + // a nullable key adds one page wholly inside its null tail, reported + // on its own row (it runs no bisection). + let valued_n = if name == "amount" { + d.count(&Query { + filter: lance_graph_quack::Filter::and([ + lance_graph_quack::Filter::plane(Mask(0)), + lance_graph_quack::Filter::plane(Mask(1)), + ]), + agg: Agg::Count, + }) + .unwrap() as usize + } else { + live_n + }; + let mut offsets = vec![0usize, valued_n / 2, valued_n - LIMIT]; + if valued_n < live_n { + offsets.push(live_n - LIMIT); + } + for offset in offsets { let mut times = Vec::new(); let (mut programs, mut materialised) = (0, 0); let mut got = Vec::new(); @@ -287,7 +306,7 @@ fn main() { ); let p = median(times); println!( - "{:<16} {:>9} {:>9} {:>12} {:>10.2} {:>12.2} {:>10.1}", + "{:<16} {:>9} {:>9} {:>12} {:>10.2} {:>12.2} {:>10.1}{}", format!( "{name} {}", if direction == Direction::Ascending { @@ -301,7 +320,12 @@ fn main() { materialised, ms(p), ms(full), - p.as_secs_f64() / scan.as_secs_f64() + p.as_secs_f64() / scan.as_secs_f64(), + if offset >= valued_n { + " (null tail)" + } else { + "" + } ); } }