diff --git a/.claude/board/entries/2026-10-21-d-qsp-0-scalar-aggregates-null.md b/.claude/board/entries/2026-10-21-d-qsp-0-scalar-aggregates-null.md index 966e18875..d014af425 100644 --- a/.claude/board/entries/2026-10-21-d-qsp-0-scalar-aggregates-null.md +++ b/.claude/board/entries/2026-10-21-d-qsp-0-scalar-aggregates-null.md @@ -62,5 +62,39 @@ Without a server the example prints `SKIP` and exits 0. It is a lab instrument; - `SUM` over a grouped key: the coalescing `Agg::GroupSumI32` still answers `0` for an unreached group (documented zero-fallback). The NULL-preserving form is `GroupReduce { GroupAgg::SumI32 }`. - A reached group whose `x` are all NULL (PostgreSQL: the group appears with `sum = NULL`) is trap T4; it belongs to the GROUP BY slice together with NULL keys (G2). - Not covered: `COALESCE`/`NULLIF`, nullable arithmetic, `int4` sum overflow past `MASKED_SUM_I32_MAX_ROWS`, `numeric` AVG. -- NULL literals and `NOT IN (…, NULL)` (G1) are not pinned here, but they need no new carrier. ⊘ Corrected: EXO-50 `/027` and `/028` said G1 needs a constant-UNKNOWN `Filter` leaf and a FALSE carrier; that was asserted without reading the substrate and is false. A column's NULLs are its presence plane (`lib.rs:18`), and `sql_where`'s leaf arm (`lib.rs:580-591`) rewrites a leaf on a nullable column to `T = valid ∧ leaf`, `F = valid ∧ ¬leaf`. A NULL operand is a borrowed alias of a lane with an all-clear presence plane, so both polarities are empty (UNKNOWN); a constant FALSE is an all-zero `Filter::Plane`. On `x = {1, 2, NULL, 1}` the existing API returns `0, 2, 2, 0, 0, 0` for `x NOT IN (1, NULL)`, its negation, `x IN (1, NULL)`, its negation, `x = NULL` and its negation, identical to PostgreSQL 16.15 (EXO-50 `/029`). The G1 rows of `entries/2026-10-21-pg-classview-polyfill-map.md` ("ABSENT") are wrong for the same reason. Open: a committed test and a random PostgreSQL differential with NULL operands. +- NULL literals and `NOT IN (…, NULL)` (G1) are not pinned here, but they need no new carrier. ⊘ Corrected: EXO-50 `/027` and `/028` said G1 needs a constant-UNKNOWN `Filter` leaf and a FALSE carrier; that was asserted without reading the substrate and is false. A column's NULLs are its presence plane (`lib.rs:18`), and `sql_where`'s leaf arm (`lib.rs:580-591`) rewrites a leaf on a nullable column to `T = valid ∧ leaf`, `F = valid ∧ ¬leaf`. A NULL operand is a borrowed alias of a lane with an all-clear presence plane, so both polarities are empty (UNKNOWN); a constant FALSE is an all-zero `Filter::Plane`. On `x = {1, 2, NULL, 1}` the existing API returns `0, 2, 2, 0, 0, 0` for `x NOT IN (1, NULL)`, its negation, `x IN (1, NULL)`, its negation, `x = NULL` and its negation, identical to PostgreSQL 16.15 (EXO-50 `/029`). The G1 rows of `entries/2026-10-21-pg-classview-polyfill-map.md` ("ABSENT") are wrong for the same reason. Now pinned (slice 2, see below). + +## Slice 2: NULL operands (G1), over existing Quack + +**Contract.** SQL NULL is absence of validity, never `0` and never FALSE. A nullable value is a value lane plus a validity plane (1 = present, 0 = NULL). For a nullable predicate, `T = valid ∧ p`, `F = valid ∧ ¬p`, UNKNOWN = `¬(T ∨ F)`. Quack's `T` is `f.sql_where(nullable)` and `F` is `Filter::Not(f).sql_where(nullable)`. +- A NULL operand (`x = NULL`, `NULL < c`, `NULL::boolean`, the NULL member of `IN`) is `Filter::Cmp(N, …)` on a column `N` whose value lane is an alias of an existing lane and whose validity plane is all clear, listed in `nullable`. Its `T` and `F` are empty on every row. +- A FALSE literal is `Filter::Plane` over an all-zero plane that is present: its `F` is every row. TRUE is `Filter::Plane` over an all-ones plane. + +**Reused, nothing added.** `Filter::{Cmp, Plane, And, Or, Not}`, `Filter::sql_where`, `lower`, `lower_fused`, mask-risc `execute_into` with `Out::Mask`, and mask-risc `Xor` then `Any` for every comparison. No new `Filter` variant, no evaluator, no counting loop, no new terminal, no change to `lib.rs`. + +**Evidence.** + +| what | result | +|---|---| +| `tests/sql_null_operand.rs`, 11 tests, 8 rows, NULL payload = the IN value | `x = NULL` and `NOT (x = NULL)` give `T = F = ∅`; `x IN (1, NULL)` gives `T` = the matches, `F = ∅`; `x NOT IN (1, NULL)` gives `T = ∅`; full AND / OR / NOT tables over TRUE, FALSE, UNKNOWN; FALSE ≠ UNKNOWN on the `F` plane; scrambling every invalid payload and the N lane changes no plane; both lowerings agree | +| `examples/qsp_null_operand_pg_parity.rs`, PostgreSQL 16.15, 2000 random expressions × 200 rows | `T` and `F` planes equal PostgreSQL on every expression in both lowerings, and again with every invalid payload re-drawn; 165278 UNKNOWN row-verdicts, 1369 expressions with a NULL operand, 250 `NOT IN (…, NULL)` with an empty `T` | +| built-in disable arms of the example | UNKNOWN read as FALSE differs from PostgreSQL on 1050 expressions; N left out of `nullable` on 1107 (both asserted > 0) | +| quack crate | 137 tests pass; clippy `-D warnings` clean | + +Disable runs, each red at an assertion line: + +- D1 (test: NULL operand mapped to the FALSE plane): 9 of 11 tests red; +- D2 (test: N not listed in `nullable`): 9 of 11 red; +- D3 (test: `sql_where` skipped, two-valued): 9 of 11 red; +- D4 (production, reverted after the run: `sql_where`'s leaf arm without its validity gate): 9 of 11 tests red, and the PostgreSQL example exits 101 on the first expression that reads a NULL payload. + +**Reproduce.** + +```text +cargo test -p lance-graph-quack --test sql_null_operand +PGHOST= PGPORT= PGUSER= PGDATABASE=postgres \ + cargo run --release -p lance-graph-quack --example qsp_null_operand_pg_parity +``` + +**Still open.** `x IS NOT DISTINCT FROM y` and other column-versus-column comparisons (G4); `COALESCE`, `NULLIF` and nullable arithmetic (value expressions); the NULL branch of a `NOT IN` subquery; the stale G1 rows in `entries/2026-10-21-pg-classview-polyfill-map.md`, which belong to their author's session. - Two folds cost two passes over the filter, as for AVG. A fused sum+count sink is a cost question, not a correctness one; no timing was taken here. diff --git a/crates/lance-graph-quack/examples/qsp_null_operand_pg_parity.rs b/crates/lance-graph-quack/examples/qsp_null_operand_pg_parity.rs new file mode 100644 index 000000000..cb9dc469b --- /dev/null +++ b/crates/lance-graph-quack/examples/qsp_null_operand_pg_parity.rs @@ -0,0 +1,561 @@ +//! D-QSP: SQL NULL operands and boolean literals, Quack against PostgreSQL, +//! row for row. +//! +//! Random expressions with NULL operands (`x = NULL`, `NULL < c`, `NULL::boolean`, +//! `x IN (.., NULL)`, `x NOT IN (.., NULL)`) over nullable columns are mapped to +//! Quack [`Filter`]s. A NULL operand is a column `N` whose value lane borrows +//! another lane and whose validity plane is all clear, listed as nullable; the +//! TRUE and FALSE literals are all-ones and all-zero planes. Each expression +//! runs through [`Filter::sql_where`] -> [`lower`] / [`lower_fused`] -> mask-risc +//! [`execute_into`] with `Out::Mask`, once for the TRUE plane (`e`) and once for +//! the FALSE plane (`NOT e`). +//! +//! PostgreSQL is the reference: it returns the TRUE rows and the FALSE rows as +//! two bit strings in `id` order. Planes are compared by mask-risc `Xor` then the +//! `Any` terminal, so an UNKNOWN row (in neither plane) cannot hide. +//! +//! Three more arms: +//! - a payload scramble: every NULL payload of `x`, `y` and the whole `N` lane +//! re-drawn, the planes must not move; +//! - disable arm 1, `unknown_as_false`: NULL operands mapped to the FALSE plane; +//! - disable arm 2, `null_operand_not_nullable`: `N` left out of the nullable +//! list. +//! +//! Both disable arms must differ from PostgreSQL on some expressions, which +//! shows the differential can see each error. +//! +//! Needs `psql` and a server, connection from the usual `PG*` variables. +//! Without one it prints `SKIP` and exits 0, so nothing here gates CI. +//! +//! ```text +//! PGHOST= PGPORT= PGUSER= PGDATABASE=postgres \ +//! cargo run --release -p lance-graph-quack --example qsp_null_operand_pg_parity +//! ``` + +use lance_graph_mask_risc::{ + execute_into, words_for, Foreign, LaneRef, MaskOp, Operand, Out, Planes, Program, Scratch, + Terminal, Value, +}; +use lance_graph_quack::{lower, lower_fused, Agg, Cmp, Col, Filter, Mask, Query}; +use std::io::Write; +use std::process::{Command, Stdio}; + +const ROWS: usize = 200; +const EXPRS: usize = 2000; + +const X: Col = Col(0); +const Y: Col = Col(1); +const N: Col = Col(2); +const VX: Mask = Mask(0); +const VY: Mask = Mask(1); +const VN: Mask = Mask(2); +const TRUE_M: Mask = Mask(3); +const FALSE_M: Mask = Mask(4); +const NULLABLE: [(Col, Mask); 3] = [(X, VX), (Y, VY), (N, VN)]; +const NULLABLE_NO_N: [(Col, Mask); 2] = [(X, VX), (Y, VY)]; + +struct Rng(u64); +impl Rng { + fn next(&mut self) -> u64 { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + self.0 + } + fn below(&mut self, n: u64) -> u64 { + self.next() % n + } +} + +/// `n` is `None` when N borrows X's lane, `Some` for an independent lane. +struct Table { + x: Vec, + y: Vec, + n: Option>, + planes: [Vec; 5], +} + +fn set(p: &mut [u64], r: usize) { + p[r / 64] |= 1 << (r % 64); +} + +fn bit(p: &[u64], r: usize) -> bool { + p[r / 64] >> (r % 64) & 1 == 1 +} + +/// Values in -4..4; the NULL payload is drawn from the same range so it +/// collides with the constants the predicates test. +fn table(rng: &mut Rng) -> Table { + let w = words_for(ROWS); + let mut t = Table { + x: vec![0; ROWS], + y: vec![0; ROWS], + n: None, + planes: std::array::from_fn(|_| vec![0; w]), + }; + for r in 0..ROWS { + for l in 0..2 { + let v = rng.below(9) as i32 - 4; + if l == 0 { + t.x[r] = v; + } else { + t.y[r] = v; + } + if rng.below(4) != 0 { + set(&mut t.planes[l], r); + } + } + set(&mut t.planes[TRUE_M.0 as usize], r); + } + t +} + +/// Same validity, every invalid payload of x and y re-drawn from a wide range, +/// and N an independent lane of arbitrary values. +fn scramble(t: &Table, rng: &mut Rng) -> Table { + const WIDE: [i32; 6] = [i32::MIN, i32::MAX, -7, 0, 1, 2]; + let mut s = Table { + x: t.x.clone(), + y: t.y.clone(), + n: Some((0..ROWS).map(|_| rng.next() as i32).collect()), + planes: t.planes.clone(), + }; + for r in 0..ROWS { + if !bit(&t.planes[0], r) { + s.x[r] = WIDE[rng.below(6) as usize]; + } + if !bit(&t.planes[1], r) { + s.y[r] = WIDE[rng.below(6) as usize]; + } + } + s +} + +#[derive(Clone, Copy)] +enum Op { + Eq, + Ne, + Lt, + Le, + Gt, + Ge, +} + +const OPS: [Op; 6] = [Op::Eq, Op::Ne, Op::Lt, Op::Le, Op::Gt, Op::Ge]; + +fn op_sql(op: Op) -> &'static str { + match op { + Op::Eq => "=", + Op::Ne => "<>", + Op::Lt => "<", + Op::Le => "<=", + Op::Gt => ">", + Op::Ge => ">=", + } +} + +/// A local expression, used only to map to a Quack filter and to SQL. +#[derive(Debug, Clone)] +enum E { + Cmp { + col: u8, + op: Op, + c: i32, + }, + CmpNull { + col: u8, + op: Op, + }, + NullCmp { + op: Op, + c: i32, + }, + True, + False, + Unknown, + In { + col: u8, + set: Vec, + has_null: bool, + }, + NotIn { + col: u8, + set: Vec, + has_null: bool, + }, + Not(Box), + And(Vec), + Or(Vec), +} + +impl std::fmt::Debug for Op { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(op_sql(*self)) + } +} + +fn col_of(c: u8) -> Col { + if c == 0 { + X + } else { + Y + } +} + +fn col_name(c: u8) -> &'static str { + if c == 0 { + "x" + } else { + "y" + } +} + +fn leaf(rng: &mut Rng) -> E { + let col = rng.below(2) as u8; + let c = rng.below(9) as i32 - 4; + let op = OPS[rng.below(6) as usize]; + let members = |rng: &mut Rng| -> Vec { + (0..1 + rng.below(3)) + .map(|_| rng.below(9) as i32 - 4) + .collect() + }; + match rng.below(10) { + 0..=2 | 8 | 9 => E::Cmp { col, op, c }, + 3 => E::CmpNull { col, op }, + 4 => E::NullCmp { op, c }, + 5 => match rng.below(3) { + 0 => E::True, + 1 => E::False, + _ => E::Unknown, + }, + 6 => E::In { + col, + set: members(rng), + has_null: rng.below(2) == 0, + }, + _ => E::NotIn { + col, + set: members(rng), + has_null: rng.below(2) == 0, + }, + } +} + +fn expr(rng: &mut Rng, depth: u32) -> E { + if depth == 0 || rng.below(4) == 0 { + return leaf(rng); + } + match rng.below(3) { + 0 => E::Not(Box::new(expr(rng, depth - 1))), + k => { + let cs = (0..1 + rng.below(4)) + .map(|_| expr(rng, depth - 1)) + .collect(); + if k == 1 { + E::And(cs) + } else { + E::Or(cs) + } + } + } +} + +fn cmp_i32(op: Op, c: i32) -> Cmp { + match op { + Op::Eq => Cmp::EqI32(c), + Op::Ne => Cmp::NeI32(c), + Op::Lt => Cmp::LtI32(c), + Op::Le => Cmp::LeI32(c), + Op::Gt => Cmp::GtI32(c), + Op::Ge => Cmp::GeI32(c), + } +} + +/// The expression as a Quack filter; `null` is the leaf every NULL operand +/// maps to (the N column in the real mapping, the FALSE plane in the disable +/// arm). A mapping only: every operator below is one Quack already has. +fn to_filter(e: &E, null: &Filter) -> Filter { + let in_list = |col: u8, set: &[i32], has_null: bool| { + let mut parts: Vec = set + .iter() + .map(|&c| Filter::Cmp(col_of(col), Cmp::EqI32(c))) + .collect(); + if has_null { + parts.push(null.clone()); + } + Filter::Or(parts) + }; + match e { + E::Cmp { col, op, c } => Filter::Cmp(col_of(*col), cmp_i32(*op, *c)), + E::CmpNull { .. } | E::NullCmp { .. } | E::Unknown => null.clone(), + E::True => Filter::Plane(TRUE_M), + E::False => Filter::Plane(FALSE_M), + E::In { col, set, has_null } => in_list(*col, set, *has_null), + E::NotIn { col, set, has_null } => Filter::Not(Box::new(in_list(*col, set, *has_null))), + E::Not(c) => Filter::Not(Box::new(to_filter(c, null))), + E::And(cs) => Filter::And(cs.iter().map(|c| to_filter(c, null)).collect()), + E::Or(cs) => Filter::Or(cs.iter().map(|c| to_filter(c, null)).collect()), + } +} + +fn sql(e: &E) -> String { + let list = |col: u8, set: &[i32], has_null: bool, kw: &str| { + let mut items: Vec = set.iter().map(|c| format!("({c})::int")).collect(); + if has_null { + items.push("NULL::int".into()); + } + format!("({} {kw} ({}))", col_name(col), items.join(", ")) + }; + match e { + E::Cmp { col, op, c } => format!("({} {} ({c})::int)", col_name(*col), op_sql(*op)), + E::CmpNull { col, op } => format!("({} {} NULL::int)", col_name(*col), op_sql(*op)), + E::NullCmp { op, c } => format!("(NULL::int {} ({c})::int)", op_sql(*op)), + E::True => "TRUE".into(), + E::False => "FALSE".into(), + E::Unknown => "NULL::boolean".into(), + E::In { col, set, has_null } => list(*col, set, *has_null, "IN"), + E::NotIn { col, set, has_null } => list(*col, set, *has_null, "NOT IN"), + E::Not(c) => format!("(NOT {})", sql(c)), + E::And(cs) => format!("({})", cs.iter().map(sql).collect::>().join(" AND ")), + E::Or(cs) => format!("({})", cs.iter().map(sql).collect::>().join(" OR ")), + } +} + +fn has_null_operand(e: &E) -> bool { + match e { + E::CmpNull { .. } | E::NullCmp { .. } | E::Unknown => true, + E::In { has_null, .. } | E::NotIn { has_null, .. } => *has_null, + E::Not(c) => has_null_operand(c), + E::And(cs) | E::Or(cs) => cs.iter().any(has_null_operand), + _ => false, + } +} + +fn has_not_in_null(e: &E) -> bool { + match e { + E::NotIn { has_null, .. } => *has_null, + E::Not(c) => has_not_in_null(c), + E::And(cs) | E::Or(cs) => cs.iter().any(has_not_in_null), + _ => false, + } +} + +#[derive(Clone, Copy)] +enum Mode { + InPlace, + Fused, +} + +fn keep(t: &Table, f: &Filter, mode: Mode, nullable: &[(Col, Mask)]) -> Vec { + let q = Query { + filter: f.sql_where(nullable), + agg: Agg::Rows, + }; + let p = match mode { + Mode::InPlace => lower(&q), + Mode::Fused => lower_fused(&q), + } + .expect("lowers"); + let masks: Vec<&[u64]> = t.planes.iter().map(Vec::as_slice).collect(); + let n_lane: &[i32] = t.n.as_deref().unwrap_or(&t.x); + let lanes = [LaneRef::I32(&t.x), LaneRef::I32(&t.y), LaneRef::I32(n_lane)]; + let pl = Planes { + n_rows: ROWS, + masks: &masks, + lanes: &lanes, + }; + let mut out = vec![0u64; words_for(ROWS)]; + let mut s = Scratch::for_program(&p, ROWS).expect("carves"); + execute_into(&p, &pl, &Foreign::NONE, &mut s, Out::Mask(&mut out)).expect("runs"); + out +} + +/// Whether two masks differ in any row: mask-risc `Xor`, then `Any`. +fn differ(a: &[u64], b: &[u64]) -> bool { + let p = Program::new( + vec![MaskOp::Xor { + a: Operand::Plane(0), + b: Operand::Plane(1), + dst: 0, + }], + Terminal::Any { + mask: Operand::Scratch(0), + }, + ); + let pl = Planes { + n_rows: ROWS, + masks: &[a, b], + lanes: &[], + }; + let mut s = Scratch::for_program(&p, ROWS).expect("carves"); + match execute_into(&p, &pl, &Foreign::NONE, &mut s, Out::None).expect("runs") { + Value::Bool(any) => any, + v => panic!("Any returned {v:?}"), + } +} + +/// PostgreSQL's bit string (`'1'` per kept row, in `id` order) as a plane. +fn plane(bits: &str) -> Vec { + assert_eq!(bits.len(), ROWS, "one character per row"); + let mut p = vec![0u64; words_for(ROWS)]; + for (r, c) in bits.bytes().enumerate() { + if c == b'1' { + set(&mut p, r); + } + } + p +} + +fn lit(valid: bool, v: impl std::fmt::Display) -> String { + if valid { + v.to_string() + } else { + "NULL".into() + } +} + +fn psql(script: String) -> Result, String> { + let out = Command::new("psql") + .args(["-X", "-q", "-tA", "-F,", "-v", "ON_ERROR_STOP=1", "-f", "-"]) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .and_then(|mut child| { + // psql answers while it reads: write on a separate thread so a + // full stdout pipe cannot block the writer. + let mut stdin = child.stdin.take().expect("piped stdin"); + let writer = std::thread::spawn(move || { + let _ = stdin.write_all(script.as_bytes()); + }); + let out = child.wait_with_output(); + let _ = writer.join(); + out + }) + .map_err(|e| format!("could not run psql ({e})"))?; + if out.status.success() { + Ok(out.stdout) + } else { + Err(format!( + "psql failed: {}", + String::from_utf8_lossy(&out.stderr).trim() + )) + } +} + +fn main() { + let mut rng = Rng(0x00D1_5EED_0007_0001); + let t = table(&mut rng); + let t2 = scramble(&t, &mut rng); + let exprs: Vec = (0..EXPRS).map(|_| expr(&mut rng, 4)).collect(); + + let mut script = + String::from("CREATE TEMP TABLE t (id int, x int, y int);\nINSERT INTO t VALUES\n"); + let values: Vec = (0..ROWS) + .map(|r| { + format!( + "({r}, {}, {})", + lit(bit(&t.planes[0], r), t.x[r]), + lit(bit(&t.planes[1], r), t.y[r]), + ) + }) + .collect(); + script.push_str(&values.join(",\n")); + script.push_str(";\n"); + for e in &exprs { + let w = sql(e); + script.push_str(&format!( + "SELECT string_agg(CASE WHEN {w} THEN '1' ELSE '0' END, '' ORDER BY id), \ + string_agg(CASE WHEN NOT {w} THEN '1' ELSE '0' END, '' ORDER BY id) FROM t;\n" + )); + } + let stdout = match psql(script) { + Ok(o) => o, + Err(why) => { + println!("SKIP: {why}"); + return; + } + }; + let lines: Vec<&str> = std::str::from_utf8(&stdout) + .expect("utf-8") + .lines() + .filter(|l| !l.is_empty()) + .collect(); + assert_eq!(lines.len(), EXPRS, "one result row per expression"); + + let real_null = Filter::Cmp(N, Cmp::EqI32(0)); + let false_null = Filter::Plane(FALSE_M); + let (mut unknown_rows, mut with_null, mut empty_not_in_null) = (0usize, 0usize, 0usize); + let (mut dis_false, mut dis_nullable) = (0usize, 0usize); + for (e, line) in exprs.iter().zip(&lines) { + let (tbits, fbits) = line.split_once(',').expect("two columns"); + let (pg_t, pg_f) = (plane(tbits), plane(fbits)); + unknown_rows += ROWS - tbits.matches('1').count() - fbits.matches('1').count(); + with_null += usize::from(has_null_operand(e)); + empty_not_in_null += usize::from(has_not_in_null(e) && !tbits.contains('1')); + + let f = to_filter(e, &real_null); + let neg = Filter::Not(Box::new(f.clone())); + for mode in [Mode::InPlace, Mode::Fused] { + assert!( + !differ(&keep(&t, &f, mode, &NULLABLE), &pg_t), + "TRUE rows differ from PostgreSQL: {e:?} = {}", + sql(e) + ); + assert!( + !differ(&keep(&t, &neg, mode, &NULLABLE), &pg_f), + "FALSE rows differ from PostgreSQL: {e:?} = {}", + sql(e) + ); + } + // Payload scramble: invalid payloads and the N lane re-drawn. + assert!( + !differ(&keep(&t2, &f, Mode::InPlace, &NULLABLE), &pg_t), + "TRUE rows depend on NULL payloads: {e:?}" + ); + assert!( + !differ(&keep(&t2, &neg, Mode::InPlace, &NULLABLE), &pg_f), + "FALSE rows depend on NULL payloads: {e:?}" + ); + + // Disable arm 1: UNKNOWN read as FALSE. + let fw = to_filter(e, &false_null); + let nw = Filter::Not(Box::new(fw.clone())); + dis_false += usize::from( + differ(&keep(&t, &fw, Mode::InPlace, &NULLABLE), &pg_t) + || differ(&keep(&t, &nw, Mode::InPlace, &NULLABLE), &pg_f), + ); + // Disable arm 2: N not listed as nullable. + dis_nullable += usize::from( + differ(&keep(&t, &f, Mode::InPlace, &NULLABLE_NO_N), &pg_t) + || differ(&keep(&t, &neg, Mode::InPlace, &NULLABLE_NO_N), &pg_f), + ); + } + let version = Command::new("psql") + .args(["-X", "-tA", "-c", "SHOW server_version"]) + .output() + .map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()) + .unwrap_or_default(); + println!( + "PostgreSQL {version}: {EXPRS} expressions over {ROWS} rows ({unknown_rows} UNKNOWN \ + row-verdicts, {with_null} with a NULL operand, {empty_not_in_null} NOT IN-with-NULL \ + with an empty TRUE plane). Quack sql_where -> lower / lower_fused -> mask-risc matches \ + PostgreSQL's TRUE and FALSE rows exactly, also with every NULL payload scrambled. \ + Disabled: unknown_as_false differs on {dis_false}, null_operand_not_nullable on \ + {dis_nullable}." + ); + assert!(unknown_rows > 0, "the table never produced UNKNOWN"); + assert!( + with_null > EXPRS / 4, + "too few expressions carry a NULL operand" + ); + assert!( + empty_not_in_null > 0, + "no NOT IN with NULL produced an empty TRUE plane" + ); + assert!(dis_false > 0, "unknown_as_false was not detected"); + assert!( + dis_nullable > 0, + "null_operand_not_nullable was not detected" + ); +} diff --git a/crates/lance-graph-quack/tests/sql_null_operand.rs b/crates/lance-graph-quack/tests/sql_null_operand.rs new file mode 100644 index 000000000..a6b5d01bf --- /dev/null +++ b/crates/lance-graph-quack/tests/sql_null_operand.rs @@ -0,0 +1,445 @@ +//! SQL NULL operands and boolean literals through Quack -> mask-risc. +//! +//! The contract. A nullable value is a value lane plus a validity plane (bit 1 +//! = present, 0 = NULL). For a nullable predicate `p`: TRUE = valid AND p, +//! FALSE = valid AND NOT p, UNKNOWN = NOT (TRUE OR FALSE). Quack computes TRUE +//! as `f.sql_where(&NULLABLE)` and FALSE as +//! `Filter::Not(f).sql_where(&NULLABLE)`. +//! +//! A SQL literal NULL operand is a column `N` whose value lane is a borrowed +//! alias of an existing lane and whose validity plane is all clear, listed in +//! `NULLABLE`. So `x = NULL`, `NULL = c` and the boolean literal `NULL::bool` +//! are all `Filter::Cmp(N, Cmp::EqI32(0))` (the constant is irrelevant: the +//! gate is clear on every row). The TRUE literal is a plane with every row set; +//! the FALSE literal is a present, all-zero plane, so FALSE and UNKNOWN stay +//! distinct. +//! +//! Every answer runs Quack `sql_where` -> `lower` / `lower_fused` -> mask-risc +//! `execute_into` with `Out::Mask`. Planes are compared by mask-risc `Xor` +//! followed by the `Any` terminal. The expected planes below are row lists read +//! off SQL semantics, not computed by an evaluator. + +use lance_graph_mask_risc::{ + execute_into, words_for, Foreign, LaneRef, MaskOp, Operand, Out, Planes, Program, Scratch, + Terminal, Value, +}; +use lance_graph_quack::{lower, lower_fused, Agg, Cmp, Col, Filter, Mask, Query}; + +const ROWS: usize = 8; + +const X: Col = Col(0); +const Y: Col = Col(1); +const N: Col = Col(2); +const VX: Mask = Mask(0); +const VY: Mask = Mask(1); +const VN: Mask = Mask(2); +const TRUE_M: Mask = Mask(3); +const FALSE_M: Mask = Mask(4); +const NULLABLE: [(Col, Mask); 3] = [(X, VX), (Y, VY), (N, VN)]; + +const ALL: [usize; ROWS] = [0, 1, 2, 3, 4, 5, 6, 7]; +const NONE: [usize; 0] = []; + +/// Resident lanes and planes. `n` is `None` when N borrows X's lane (the same +/// slice passed twice); `Some` when a test gives it arbitrary values. +struct Table { + x: Vec, + y: Vec, + n: Option>, + planes: [Vec; 5], +} + +fn rows(rs: &[usize]) -> Vec { + let mut p = vec![0u64; words_for(ROWS)]; + for &r in rs { + p[r / 64] |= 1 << (r % 64); + } + p +} + +/// x = [1, 2, NULL, 1, NULL, 3, 0, 1], y = [NULL, 1, 1, NULL, 0, 2, NULL, 5]; +/// every NULL row's payload is 1 (the IN value, so a payload read is caught). +fn base() -> Table { + Table { + x: vec![1, 2, 1, 1, 1, 3, 0, 1], + y: vec![1, 1, 1, 1, 0, 2, 1, 5], + n: None, + planes: [ + rows(&[0, 1, 3, 5, 6, 7]), + rows(&[1, 2, 4, 5, 7]), + rows(&NONE), + rows(&ALL), + rows(&NONE), + ], + } +} + +/// The same validity, every NULL payload changed, N over arbitrary values. +fn scrambled() -> Table { + let mut t = base(); + t.x = vec![1, 2, 0, 1, i32::MIN, 3, 0, 1]; + t.y = vec![2, 1, 1, -7, 0, 2, i32::MAX, 5]; + t.n = Some(vec![5, -1, 0, 1, 1, i32::MIN, 0, i32::MAX]); + t +} + +#[derive(Clone, Copy)] +enum Mode { + InPlace, + Fused, +} + +fn keep(t: &Table, f: &Filter, mode: Mode, nullable: &[(Col, Mask)]) -> Vec { + let q = Query { + filter: f.sql_where(nullable), + agg: Agg::Rows, + }; + let p = match mode { + Mode::InPlace => lower(&q), + Mode::Fused => lower_fused(&q), + } + .expect("lowers"); + let masks: Vec<&[u64]> = t.planes.iter().map(Vec::as_slice).collect(); + let n_lane: &[i32] = t.n.as_deref().unwrap_or(&t.x); + let lanes = [LaneRef::I32(&t.x), LaneRef::I32(&t.y), LaneRef::I32(n_lane)]; + let pl = Planes { + n_rows: ROWS, + masks: &masks, + lanes: &lanes, + }; + let mut out = vec![0u64; words_for(ROWS)]; + let mut s = Scratch::for_program(&p, ROWS).expect("carves"); + execute_into(&p, &pl, &Foreign::NONE, &mut s, Out::Mask(&mut out)).expect("runs"); + out +} + +/// Whether two masks differ in any row: mask-risc `Xor`, then `Any`. +fn differ(a: &[u64], b: &[u64]) -> bool { + let p = Program::new( + vec![MaskOp::Xor { + a: Operand::Plane(0), + b: Operand::Plane(1), + dst: 0, + }], + Terminal::Any { + mask: Operand::Scratch(0), + }, + ); + let pl = Planes { + n_rows: ROWS, + masks: &[a, b], + lanes: &[], + }; + let mut s = Scratch::for_program(&p, ROWS).expect("carves"); + match execute_into(&p, &pl, &Foreign::NONE, &mut s, Out::None).expect("runs") { + Value::Bool(any) => any, + v => panic!("Any returned {v:?}"), + } +} + +/// (TRUE plane, FALSE plane) of `f` on `t` under `nullable`, for both +/// lowerings, which must agree. +fn planes_with(t: &Table, f: &Filter, nullable: &[(Col, Mask)]) -> (Vec, Vec) { + let neg = Filter::Not(Box::new(f.clone())); + let (ti, fi) = ( + keep(t, f, Mode::InPlace, nullable), + keep(t, &neg, Mode::InPlace, nullable), + ); + let (tf_, ff) = ( + keep(t, f, Mode::Fused, nullable), + keep(t, &neg, Mode::Fused, nullable), + ); + assert!( + !differ(&ti, &tf_), + "TRUE plane: lower and lower_fused disagree for {f:?}" + ); + assert!( + !differ(&fi, &ff), + "FALSE plane: lower and lower_fused disagree for {f:?}" + ); + (ti, fi) +} + +fn tf(f: &Filter) -> (Vec, Vec) { + planes_with(&base(), f, &NULLABLE) +} + +fn assert_k3(f: &Filter, expect_t: &[usize], expect_f: &[usize], label: &str) { + let (t, fl) = tf(f); + assert!( + !differ(&t, &rows(expect_t)), + "{label}: TRUE plane differs from the expected rows {expect_t:?}" + ); + assert!( + !differ(&fl, &rows(expect_f)), + "{label}: FALSE plane differs from the expected rows {expect_f:?}" + ); +} + +// ----- mappings (leaves Quack already has) ----- + +fn null_leaf() -> Filter { + Filter::Cmp(N, Cmp::EqI32(0)) +} +fn x_eq(c: i32) -> Filter { + Filter::Cmp(X, Cmp::EqI32(c)) +} +fn x_in(set: &[i32], null: Option) -> Filter { + let mut parts: Vec = set.iter().map(|&c| x_eq(c)).collect(); + parts.extend(null); + Filter::Or(parts) +} +fn not(f: Filter) -> Filter { + Filter::Not(Box::new(f)) +} +fn tru() -> Filter { + Filter::Plane(TRUE_M) +} +fn fals() -> Filter { + Filter::Plane(FALSE_M) +} + +/// FAILS IF `x = NULL` is read as FALSE (F would be every row). +#[test] +fn x_eq_null_is_unknown_not_false() { + assert_k3(&null_leaf(), &NONE, &NONE, "x = NULL"); +} + +/// FAILS IF NOT over UNKNOWN turns TRUE (T would be every row). +#[test] +fn not_x_eq_null_stays_unknown_not_true() { + assert_k3(¬(null_leaf()), &NONE, &NONE, "NOT (x = NULL)"); +} + +/// FAILS IF a NULL IN-member does not move the non-matches from FALSE to +/// UNKNOWN, or a NULL row of x is treated as present. +#[test] +fn x_in_1_null_splits_match_nonmatch_and_null() { + assert_k3( + &x_in(&[1], Some(null_leaf())), + &[0, 3, 7], + &NONE, + "x IN (1, NULL)", + ); + assert_k3(&x_in(&[1, 2], None), &[0, 1, 3, 7], &[5, 6], "x IN (1, 2)"); +} + +/// FAILS IF `NOT IN` with a NULL member lets an UNKNOWN row through as TRUE. +#[test] +fn x_not_in_1_null_never_turns_unknown_true() { + assert_k3( + ¬(x_in(&[1], Some(null_leaf()))), + &NONE, + &[0, 3, 7], + "x NOT IN (1, NULL)", + ); + assert_k3( + ¬(x_in(&[1, 2], None)), + &[5, 6], + &[0, 1, 3, 7], + "x NOT IN (1, 2)", + ); +} + +#[derive(Clone, Copy)] +enum K { + T, + F, + U, +} + +fn konst(k: K) -> Filter { + match k { + K::T => tru(), + K::F => fals(), + K::U => null_leaf(), + } +} + +fn assert_const(f: &Filter, expect: K, label: &str) { + match expect { + K::T => assert_k3(f, &ALL, &NONE, label), + K::F => assert_k3(f, &NONE, &ALL, label), + K::U => assert_k3(f, &NONE, &NONE, label), + } +} + +/// FAILS IF AND is two-valued: FALSE AND UNKNOWN must be FALSE (either order), +/// TRUE AND UNKNOWN must be UNKNOWN. +#[test] +fn k3_and_truth_table() { + use K::{F, T, U}; + let table = [ + (T, T, T), + (T, F, F), + (T, U, U), + (F, T, F), + (F, F, F), + (F, U, F), + (U, T, U), + (U, F, F), + (U, U, U), + ]; + for (a, b, r) in table { + let f = Filter::And(vec![konst(a), konst(b)]); + assert_const(&f, r, "AND"); + } +} + +/// FAILS IF OR is two-valued: TRUE OR UNKNOWN must be TRUE (either order), +/// FALSE OR UNKNOWN must be UNKNOWN. +#[test] +fn k3_or_truth_table() { + use K::{F, T, U}; + let table = [ + (T, T, T), + (T, F, T), + (T, U, T), + (F, T, T), + (F, F, F), + (F, U, U), + (U, T, T), + (U, F, U), + (U, U, U), + ]; + for (a, b, r) in table { + let f = Filter::Or(vec![konst(a), konst(b)]); + assert_const(&f, r, "OR"); + } +} + +/// FAILS IF NOT UNKNOWN is not UNKNOWN, or NOT flips a present constant wrongly. +#[test] +fn k3_not_truth_table() { + use K::{F, T, U}; + for (a, r) in [(T, F), (F, T), (U, U)] { + assert_const(¬(konst(a)), r, "NOT"); + } +} + +/// FAILS IF the FALSE literal and UNKNOWN collapse into one verdict. +#[test] +fn false_literal_and_unknown_are_distinct() { + let (tf_, ff) = tf(&fals()); + assert!( + !differ(&tf_, &rows(&NONE)), + "FALSE literal: TRUE plane must be empty" + ); + assert!( + !differ(&ff, &rows(&ALL)), + "FALSE literal: FALSE plane must be every row" + ); + let (tu, fu) = tf(&null_leaf()); + assert!( + !differ(&tu, &rows(&NONE)), + "UNKNOWN: TRUE plane must be empty" + ); + assert!( + !differ(&fu, &rows(&NONE)), + "UNKNOWN: FALSE plane must be empty" + ); + assert!( + differ(&ff, &fu), + "FALSE and UNKNOWN must differ on the FALSE plane" + ); +} + +fn battery() -> Vec<(&'static str, Filter)> { + let x_in_1_null = x_in(&[1], Some(null_leaf())); + let y_in = Filter::Or(vec![Filter::Cmp(Y, Cmp::EqI32(1)), null_leaf()]); + vec![ + ("x = NULL", null_leaf()), + ("NOT x = NULL", not(null_leaf())), + ("x IN (1, NULL)", x_in_1_null.clone()), + ("x NOT IN (1, NULL)", not(x_in_1_null)), + ("x IN (1, 2)", x_in(&[1, 2], None)), + ("x NOT IN (1, 2)", not(x_in(&[1, 2], None))), + ("x = 1", x_eq(1)), + ("x <> 1", Filter::Cmp(X, Cmp::NeI32(1))), + ("y > 0", Filter::Cmp(Y, Cmp::GtI32(0))), + ( + "x = 1 AND y = NULL", + Filter::And(vec![x_eq(1), null_leaf()]), + ), + ( + "(x = 1 OR y = NULL) AND NOT (y IN (1, NULL))", + Filter::And(vec![Filter::Or(vec![x_eq(1), null_leaf()]), not(y_in)]), + ), + ( + "TRUE AND NOT (x IN (0, NULL))", + Filter::And(vec![tru(), not(x_in(&[0], Some(null_leaf())))]), + ), + ] +} + +/// FAILS IF any verdict reads the payload of a NULL row of x or y, or the +/// values of the N alias lane. +#[test] +fn invalid_payload_changes_nothing() { + let (a, b) = (base(), scrambled()); + for (label, f) in battery() { + let (ta, fa) = planes_with(&a, &f, &NULLABLE); + let (tb, fb) = planes_with(&b, &f, &NULLABLE); + assert!( + !differ(&ta, &tb), + "{label}: TRUE plane depends on NULL payloads" + ); + assert!( + !differ(&fa, &fb), + "{label}: FALSE plane depends on NULL payloads" + ); + } +} + +/// FAILS IF the checker cannot tell UNKNOWN read as FALSE from UNKNOWN: with +/// every NULL operand mapped to the FALSE plane, NOT (x = NULL) becomes TRUE on +/// every row and x NOT IN (1, NULL) keeps rows 1, 5, 6. +#[test] +fn unknown_read_as_false_is_detected() { + let wrong_not = not(fals()); + let (t, _) = tf(&wrong_not); + assert!( + !differ(&t, &rows(&ALL)), + "the wrong reading is TRUE on every row" + ); + assert!( + differ(&t, &rows(&NONE)), + "NOT (x = NULL): wrong reading must differ from []" + ); + + let wrong_not_in = not(x_in(&[1], Some(fals()))); + let (t, _) = tf(&wrong_not_in); + assert!( + !differ(&t, &rows(&[1, 5, 6])), + "the wrong reading keeps rows 1, 5, 6" + ); + assert!( + differ(&t, &rows(&NONE)), + "x NOT IN (1, NULL): wrong reading must differ from []" + ); +} + +/// FAILS IF the checker cannot tell a NULL operand left out of the nullable +/// list (its alias payload read two-valued) from the correct three-valued one. +#[test] +fn null_operand_left_out_of_nullable_is_detected() { + let without_n = [(X, VX), (Y, VY)]; + let (t, f) = planes_with(&base(), &null_leaf(), &without_n); + assert!( + differ(&t, &rows(&NONE)), + "x = NULL: TRUE plane must differ without N listed" + ); + assert!( + differ(&f, &rows(&NONE)), + "x = NULL: FALSE plane must differ without N listed" + ); + + let e = not(x_in(&[1], Some(null_leaf()))); + let (t, f) = planes_with(&base(), &e, &without_n); + let diff_t = differ(&t, &rows(&NONE)); + let diff_f = differ(&f, &rows(&[0, 3, 7])); + assert!( + diff_t || diff_f, + "x NOT IN (1, NULL): some plane must differ without N listed" + ); +}