Probe P-VIEW-1: saved-view page cost at 1M and 4M rows

.claude/board/ISSUES.md

@@ -3,6 +3,21 @@
 > Open blockers and cross-repo follow-ups. Append new entries at the top;
 > change only the `Status` line of an existing entry.
 
+## ISS-ORDER-BY-RANK-SELECT — a page sorted by an `I32` key costs more than sorting everything
+
+- **Status:** OPEN 2026-10-21. Measured by P-VIEW-1; the ask is posted to lance-graph on EXO-50 (operator's instruction: probe first, then post with numbers). HubSPO-rs builds nothing in place of it.
+- **Where:** `hubspo-query::view::page`, the `I32` arm (`value_at` bisection).
+- **What:** without a rank-select or `ORDER BY` terminal, the page's first and last value are found by bisection: 67 `Count` programs for every page that has valued records, at any offset. A page wholly inside a nullable key's null tail runs 2 programs and no bisection (Codex on #46: the first run's `amount` tail page was such a page). Measured (`cargo run --release -p hubspo-query --example probe_view_page`, 25-record pages, median of 5 per run, ranges over three runs on a shared 4-core container):
+
+  | rows | one threshold scan | `I32` page (valued) | full edge sort | code-key page | null-tail page |
+  |---|---|---|---|---|---|
+  | 1M | 1.16–2.31 ms | 79–142 ms | 38–66 ms | 2.0–3.2 ms | 0.36–0.54 ms |
+  | 4M | 5.34–5.85 ms | 352–421 ms | 196–242 ms | 8.9–13.9 ms | 1.5–2.3 ms |
+
+  A valued `I32` page is 60–100 scans (same-run ratio) and 1.6–2.8× a full sort at the edge. A code-key page is ~1–2.7 scans but materialises the whole boundary code (~25 % of rows with 4 codes). Costs scale linearly with rows.
+- **Ask:** one program that returns the value at a given rank of an `I32` lane over a filter (rank select), or a top-k / `ORDER BY` terminal returning the ordinals of ranks `[offset, offset + k)` with ties by ordinal. Either replaces the 66 bisection programs with one scan.
+
+
 ## ISS-PRESET-VERBS-PROVISIONAL — the preset roles' verbs are written from seat names, not from HubSpot
 
 - **Status:** DECIDED 2026-10-21 (operator, EXO-50 message 006): AMENDED, not approved. The six role names stay. The verb table stays only as a tagged compatibility/simulation fixture, never as tenant runtime access. EffectiveAccess = seat/subscription ceiling ∧ tenant permission grants ∧ owner/team scope ∧ per-property outer+ordinal authorization ∧ action-specific gates. Super Admin is an explicit status, not a bypass of tenant isolation, schema/policy validation or seat gates. Admin is not blanket FULL/DELETE. View-only has no implicit read of sensitive fields. ACT (email send, enrolment, invoice, bulk), DELETE and EXPORT each need an explicit grant and are never inherited from WRITE/CREATE. A missing, unknown, stale or unread tenant grant fails closed and never falls back to the fixture. The presets are not reported as HubSpot-verified until a sandbox oracle exists (ISS-ORACLE-PORTAL). Implemented in the D-IAM-3 code PR (2026-10-21): the runtime bind grants nothing, the table is `PresetFixture`.

.claude/board/LATEST_STATE.md

@@ -217,8 +217,16 @@
   `before` count ignored, nulls dropped, bisection off by one, code order
   not reversed, band missing the view's filter, sort left out of the gate,
   hidden column kept.
-- Open: a page-cost probe at 1M+ rows (the `ORDER BY` question waits on
-  it), multi-key sorts, view sharing (owner/team/everyone) and persistence.
+- P-VIEW-1 (`examples/probe_view_page.rs`, 2026-10-21): 25-record pages
+  at 1M and 4M rows, ranges over three runs. `I32` key: 67 programs on every
+  page with valued records, 79–142 ms at 1M and 352–421 ms at 4M, 60–100×
+  one threshold scan and 1.6–2.8× sorting every kept record at the edge. A
+  page wholly in a nullable key's null tail: 2 programs. Code key: 3
+  programs, ~1–2.7 scans, but the whole boundary code is materialised.
+  Linear in rows. The probe failed for `I32`; the rank-select
+  / `ORDER BY` ask is on EXO-50 (`ISS-ORDER-BY-RANK-SELECT`).
+- Open: that ask, multi-key sorts, view sharing (owner/team/everyone) and
+  persistence.
 
 ## Lists (D-QRY-2)
 

.claude/board/STATUS_BOARD.md

@@ -49,7 +49,7 @@
 |---|---|---|---|---|
 | D-QRY-1 | Filter compiler: HubSpot filter groups (AND within group, OR across groups; operators EQ/NEQ/LT/…/HAS_PROPERTY/CONTAINS_TOKEN/IN/BETWEEN) → Quack program | L | Shipped | 2026-10-14: lance-graph #1546 `lance-graph-relalg` (D-RA-1) normalizes a filter before it runs and answers a contradictory one as `Empty` without a program; D-CMR-1 moves that set algebra into the contract. Adopt it there, not from the prototype crate; `hubspo-store::catalog` + `hubspo-query::filter`; 600 random filter sets equal a plain-loop oracle on mask-risc; #19 |
 | D-QRY-2 | Lists: active (incremental on version delta) + static (stored mask); membership history | L | Shipped | `hubspo-query::list` (refresh, static add/remove/members, added/removed); tests vs full recomputation. P-LIST-1 2.6 ms at 10M; P-SPARSE-1 1.93 ms, 257 KiB flat at 10M. Storage of list masks lands with D-STO-1 |
-| D-QRY-3 | Saved views + board view (report pivot over pipeline × stage) | M | In progress | board half shipped (#21): `hubspo-query::board` (count + amount per stage as two `GroupReduce` programs); saved-view half shipped 2026-10-21: `hubspo-query::view` (`SavedView`, `page` with edge sort: storage order, `Count`-threshold bisection for I32, `GroupReduce Count` for codes; ties by ordinal, nulls last), oracle over every page, `hubspo-compat::views::open_view` through the D-IAM-3 gate; 8 disable runs red. Open: page-cost probe at 1M+ rows (ORDER BY asked upstream only if it fails), multi-key sort, sharing, persistence |
+| D-QRY-3 | Saved views + board view (report pivot over pipeline × stage) | M | In progress | board half shipped (#21): `hubspo-query::board` (count + amount per stage as two `GroupReduce` programs); saved-view half shipped 2026-10-21: `hubspo-query::view` (`SavedView`, `page` with edge sort: storage order, `Count`-threshold bisection for I32, `GroupReduce Count` for codes; ties by ordinal, nulls last), oracle over every page, `hubspo-compat::views::open_view` through the D-IAM-3 gate; 8 disable runs red. P-VIEW-1 2026-10-21: the probe failed for `I32` keys (a valued page is 67 programs, 60–100 scans, 1.6–2.8× a full edge sort; 79–142 ms at 1M, 352–421 ms at 4M) and passed for code keys (~1–2.7 scans); a null-tail page is 2 programs; rank-select/ORDER BY asked upstream on EXO-50 (`ISS-ORDER-BY-RANK-SELECT`). Open: that ask, multi-key sort, sharing, persistence |
 | D-QRY-4 | Quack lowering of callcenter's `ParsedQuery` (the upstream DataFusion `parsed_query_to_plan` is an unimplemented stub) | M | Queued |  |
 | D-QRY-5 | Association-traversing filters ("contacts whose company has an open deal"): Quack `Semijoin` over edge lanes for one hop; multi-hop needs cypher-quack shapes that do not exist yet → upstream request first | L | Queued | 2026-10-20: lance-graph #1632/#1633 (D-SRAM-2 lab spike, `crepe-plan-probe`) measured a semijoin reduction before preparation and an addressed semijoin over attested lanes with masked Lance 13 reads; lab only, not authorization; 2026-10-14: the hop is D-CMR-5, preimage `f⁻¹(S)` under a proven mapping (lance-graph `d-cmr-concept-chain-normalization-v1`), not chained key reads (withdrawn in #1524 / ndarray #359). One hop is ready on the existing Quack `Semijoin`; multi-hop waits on D-CMR-5 rather than on new cypher-quack shapes |
 | D-QRY-6 | Reports: single-object, cross-object, funnel, time-series, attribution skeleton on `lance-graph-report` | XL | Queued | 2026-10-21 ruling: also owns forecast types (`crm-forecast-types`, 2 functions) and reads D-FLOW-2's custom behavioural events. |

crates/hubspo-query/examples/probe_view_page.rs

@@ -0,0 +1,332 @@
+//! P-VIEW-1: what one saved-view page costs without `ORDER BY`.
+//!
+//! For each sort key and offset, one 25-record page through
+//! `hubspo_query::view::page`, against two references on the same data:
+//!
+//! - **full sort:** every kept record materialised, sorted by (value,
+//!   ordinal), sliced: what a page costs if the edge sorts everything;
+//! - **one scan:** a single threshold `Agg::Count` over the key lane: a floor
+//!   for any in-engine `ORDER BY`, which has to read the key lane at least once.
+//!
+//! Reported per case: programs run, rows materialised, median wall time of
+//! `RUNS` repetitions. Keys: `score` (i32, near-unique), `amount` (i32 in
+//! -20..20, 20 % null: heavy ties), `stage` (code, 4 values: a page inside
+//! one code materialises that code).
+//!
+//! `cargo run --release -p hubspo-query --example probe_view_page [rows]`
+
+use std::time::{Duration, Instant};
+
+use hubspo_query::view::{Direction, Runner, SavedView, Sort, page};
+use hubspo_store::catalog::{Catalog, Codebook, FieldSpec};
+use lance_graph_mask_risc::{
+    Foreign, LaneRef, Out, Planes, Scratch, Value, execute_into, materialize_rows, words_for,
+};
+use lance_graph_quack::bind::{FieldKind, create_table};
+use lance_graph_quack::{Agg, Col, Mask, Query, lower};
+
+const LIMIT: usize = 25;
+const RUNS: usize = 5;
+const STAGES: &[&str] = &["lead", "opportunity", "customer", "evangelist"];
+
+struct Rng(u64);
+impl Rng {
+    fn next(&mut self) -> u64 {
+        self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
+        let mut z = self.0;
+        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+        z ^ (z >> 31)
+    }
+}
+
+struct Data {
+    n: usize,
+    live: Vec<u64>,
+    amount_valid: Vec<u64>,
+    stage: Vec<u32>,
+    amount: Vec<i32>,
+    score: Vec<i32>,
+    programs: usize,
+    materialised: usize,
+}
+
+impl Data {
+    fn new(n: usize) -> Self {
+        let mut r = Rng(0x0D3E_5EED);
+        let mut live = vec![0u64; words_for(n)];
+        let mut amount_valid = vec![0u64; words_for(n)];
+        let (mut stage, mut amount, mut score) = (Vec::new(), Vec::new(), Vec::new());
+        for i in 0..n {
+            let x = r.next();
+            if x % 100 >= 5 {
+                live[i / 64] |= 1 << (i % 64);
+            }
+            if (x >> 8) % 100 >= 20 {
+                amount_valid[i / 64] |= 1 << (i % 64);
+            }
+            stage.push(((x >> 16) % 4) as u32);
+            amount.push(((x >> 24) % 41) as i32 - 20);
+            score.push(r.next() as i32);
+        }
+        Self {
+            n,
+            live,
+            amount_valid,
+            stage,
+            amount,
+            score,
+            programs: 0,
+            materialised: 0,
+        }
+    }
+
+    fn run(&mut self, q: &Query, out: Out<'_>) -> Value {
+        self.programs += 1;
+        let masks: [&[u64]; 2] = [&self.live, &self.amount_valid];
+        let lanes = [
+            LaneRef::U32(&self.stage),
+            LaneRef::I32(&self.amount),
+            LaneRef::I32(&self.score),
+        ];
+        let planes = Planes {
+            n_rows: self.n,
+            masks: &masks,
+            lanes: &lanes,
+        };
+        let program = lower(q).expect("lower");
+        let mut scratch = Scratch::for_program(&program, self.n).expect("scratch");
+        execute_into(&program, &planes, &Foreign::NONE, &mut scratch, out).expect("execute")
+    }
+}
+
+impl Runner for Data {
+    type Error = ();
+    fn count(&mut self, q: &Query) -> Result<u64, ()> {
+        match self.run(q, Out::None) {
+            Value::Count(n) => Ok(n as u64),
+            v => panic!("{v:?}"),
+        }
+    }
+    fn rows(&mut self, q: &Query) -> Result<Vec<usize>, ()> {
+        let mut kept = vec![0u64; words_for(self.n)];
+        self.run(q, Out::Mask(&mut kept));
+        let rows = materialize_rows(&kept, self.n);
+        self.materialised += rows.len();
+        Ok(rows)
+    }
+    fn groups(&mut self, q: &Query, groups: u32) -> Result<Vec<i64>, ()> {
+        let mut sink = vec![0i64; groups as usize];
+        self.run(q, Out::I64(&mut sink));
+        Ok(sink)
+    }
+    fn code_domain(&self, _: Col) -> u32 {
+        STAGES.len() as u32
+    }
+    fn i32_at(&self, col: Col, row: usize) -> i32 {
+        match col {
+            Col(1) => self.amount[row],
+            _ => self.score[row],
+        }
+    }
+}
+
+fn catalog() -> Catalog {
+    let mut c = Catalog::new();
+    let t = create_table("contacts")
+        .width(8)
+        .register(&mut c)
+        .expect("register")
+        .id;
+    let fields = [
+        ("lifecyclestage", Col(0), FieldKind::Code, None),
+        ("amount", Col(1), FieldKind::I32, Some(Mask(1))),
+        ("score", Col(2), FieldKind::I32, None),
+    ];
+    for (name, col, kind, validity) in fields {
+        c.declare(
+            t,
+            FieldSpec {
+                name: name.into(),
+                col,
+                kind,
+                validity,
+                codebook: (kind == FieldKind::Code)
+                    .then(|| Codebook::new(STAGES.iter().copied()).expect("codebook")),
+            },
+        )
+        .expect("declare");
+    }
+    c
+}
+
+fn median(mut v: Vec<Duration>) -> Duration {
+    v.sort_unstable();
+    v[v.len() / 2]
+}
+
+fn ms(d: Duration) -> f64 {
+    d.as_secs_f64() * 1e3
+}
+
+/// Every kept record, sorted at the edge: the reference a full sort costs.
+fn full_sort(d: &mut Data, key: Col, offset: usize) -> (Vec<usize>, usize) {
+    let live = Query {
+        filter: lance_graph_quack::Filter::plane(Mask(0)),
+        agg: Agg::Rows,
+    };
+    let rows = d.rows(&live).expect("rows");
+    let n = rows.len();
+    let mut keyed: Vec<(i32, usize)> = rows
+        .into_iter()
+        .map(|r| {
+            let v = match key {
+                Col(0) => d.stage[r] as i32,
+                Col(1) => d.amount[r],
+                _ => d.score[r],
+            };
+            (v, r)
+        })
+        .collect();
+    keyed.sort_unstable();
+    (
+        keyed
+            .into_iter()
+            .skip(offset)
+            .take(LIMIT)
+            .map(|(_, r)| r)
+            .collect(),
+        n,
+    )
+}
+
+fn main() {
+    let n: usize = std::env::args()
+        .nth(1)
+        .map(|a| a.parse().expect("rows"))
+        .unwrap_or(1_000_000);
+    let cat = catalog();
+    let mut d = Data::new(n);
+    let live_n = d
+        .count(&Query {
+            filter: lance_graph_quack::Filter::plane(Mask(0)),
+            agg: Agg::Count,
+        })
+        .unwrap() as usize;
+
+    // One scan of the key lane: a threshold Count, the floor any in-engine
+    // ORDER BY pays (a bare plane Count is a popcount and reads no lane).
+    let scan = median(
+        (0..RUNS)
+            .map(|_| {
+                let t = Instant::now();
+                d.count(&Query {
+                    filter: lance_graph_quack::Filter::and([
+                        lance_graph_quack::Filter::plane(Mask(0)),
+                        lance_graph_quack::Filter::cmp(Col(2), lance_graph_quack::Cmp::LeI32(0)),
+                    ]),
+                    agg: Agg::Count,
+                })
+                .unwrap();
+                t.elapsed()
+            })
+            .collect(),
+    );
+    println!("rows {n}, live {live_n}, page {LIMIT}, median of {RUNS}");
+    println!(
+        "one scan (threshold Count on the key lane): {:.2} ms",
+        ms(scan)
+    );
+    println!(
+        "{:<16} {:>9} {:>9} {:>12} {:>10} {:>12} {:>10}",
+        "key", "offset", "programs", "materialised", "page ms", "full-sort ms", "page/scan"
+    );
+
+    let keys = [
+        ("score", Col(2), Direction::Ascending),
+        ("score", Col(2), Direction::Descending),
+        ("amount", Col(1), Direction::Ascending),
+        ("lifecyclestage", Col(0), Direction::Ascending),
+    ];
+    for (name, col, direction) in keys {
+        let view = SavedView {
+            object: "contacts".into(),
+            sort: Some(Sort {
+                property: name.into(),
+                direction,
+            }),
+            ..SavedView::default()
+        };
+        // Offsets over the key's VALUED records, so the tail page still sorts;
+        // a nullable key adds one page wholly inside its null tail, reported
+        // on its own row (it runs no bisection).
+        let valued_n = if name == "amount" {
+            d.count(&Query {
+                filter: lance_graph_quack::Filter::and([
+                    lance_graph_quack::Filter::plane(Mask(0)),
+                    lance_graph_quack::Filter::plane(Mask(1)),
+                ]),
+                agg: Agg::Count,
+            })
+            .unwrap() as usize
+        } else {
+            live_n
+        };
+        let mut offsets = vec![0usize, valued_n / 2, valued_n - LIMIT];
+        if valued_n < live_n {
+            offsets.push(live_n - LIMIT);
+        }
+        for offset in offsets {
+            let mut times = Vec::new();
+            let (mut programs, mut materialised) = (0, 0);
+            let mut got = Vec::new();
+            for _ in 0..RUNS {
+                d.programs = 0;
+                d.materialised = 0;
+                let t = Instant::now();
+                got = page(&view, &cat, offset, LIMIT, &mut d).expect("page");
+                times.push(t.elapsed());
+                programs = d.programs;
+                materialised = d.materialised;
+            }
+            assert_eq!(got.len(), LIMIT, "{name} {offset}");
+            // The full-sort reference, and a check against it where the
+            // order agrees (ascending, no nulls).
+            let full = median(
+                (0..RUNS)
+                    .map(|_| {
+                        let t = Instant::now();
+                        let (want, _) = full_sort(&mut d, col, offset);
+                        if direction == Direction::Ascending && name != "amount" {
+                            assert_eq!(got, want, "{name} {offset}");
+                        }
+                        t.elapsed()
+                    })
+                    .collect(),
+            );
+            let p = median(times);
+            println!(
+                "{:<16} {:>9} {:>9} {:>12} {:>10.2} {:>12.2} {:>10.1}{}",
+                format!(
+                    "{name} {}",
+                    if direction == Direction::Ascending {
+                        "asc"
+                    } else {
+                        "desc"
+                    }
+                ),
+                offset,
+                programs,
+                materialised,
+                ms(p),
+                ms(full),
+                p.as_secs_f64() / scan.as_secs_f64(),
+                if offset >= valued_n {
+                    "  (null tail)"
+                } else {
+                    ""
+                }
+            );
+        }
+    }
+}