diff --git a/.claude/board/entries/2026-10-05-quack-sql-null-3vl.md b/.claude/board/entries/2026-10-05-quack-sql-null-3vl.md new file mode 100644 index 000000000..25922284e --- /dev/null +++ b/.claude/board/entries/2026-10-05-quack-sql-null-3vl.md @@ -0,0 +1,70 @@ +# 2026-10-05 — Quack WHERE is SQL three-valued over nullable columns; SQL READ coverage re-evaluated + +**Status:** TEST-PINNED (`crates/lance-graph-quack/tests/sql_null_3vl.rs`, `bind::tests::a_nullable_field_binds_three_valued`) · OPEN (coverage items below) + +## DECISION — the NULL representation (operator, 2026-10-05) + +A nullable column is **an ordinary value lane plus its resident validity plane** — the separation DuckDB uses. `value 0, valid 1` is a real zero; `value 0, valid 0` is NULL. NULL never becomes a value: no boxed or tagged NULL, no second value representation, no second bitmap, no expression interpreter. + +## What landed + +`Filter::sql_where(&self, nullable: &[(Col, Mask)]) -> Filter` (plus `Filter::is_null` / `is_not_null` over the validity plane). It rewrites a filter into an ordinary two-valued filter that keeps exactly the rows SQL 3VL makes TRUE. + +- **Model:** each subformula lowers to `T(φ)` (TRUE rows) or `F(φ)` (FALSE rows); UNKNOWN = neither. Leaf on nullable `x`: `T = valid ∧ leaf`, `F = valid ∧ ¬leaf`. `NOT` swaps T/F; `AND`: `T = ⋀T`, `F = ⋁F`; `OR`: dual. `WHERE` keeps `T(root)`. This is the `(true, known)` pair in its dual-rail form: `known = T ∪ F`. +- **mask-risc unchanged.** The output uses the same `Cmp`/`Plane`/`And`/`Or`/`Not` nodes, so `lower` and `lower_fused` run it as is; `valid(x)` is an ordinary resident-plane gate the survivor skip uses. +- **Non-nullable fast path:** a filter (or subtree) reading no listed column is returned unchanged — identical program, both lowerings. Linear size. +- **Leaves:** `Cmp` nullable by column — including `Cmp::Range`, which the executor answers from the row ordinal but which stands for a comparison on its provenance lane (codex P2; the first cut exempted it and kept NULL rows inside `[lo, hi)`); `EqU32Via` by its `fk` (a NULL fk makes `f.v = c` UNKNOWN); `Plane` never (a known Boolean — which is what makes IS [NOT] NULL exact). +- **`Semijoin` is NOT three-valued.** It is `EXISTS(SELECT 1 FROM foreign f WHERE f.rid = this.fk AND …)`; with a NULL fk the inner match is empty and `EXISTS` is FALSE, a known answer. Its `Gather` still reads the foreign plane at the NULL row's stale payload, so a listed fk gates it: `T = valid ∧ leaf`, `F = ¬(valid ∧ leaf)` — `NOT semijoin` KEEPS NULL-fk rows, `NOT eq_via` does not. +- **Binder wiring.** `BoundField` gains `validity: Option` — the binder owns the schema, so it owns nullability. `Draft::bind` collects the validity of every field a predicate reads and returns `Filter::and(parts).sql_where(&nullable)`; with no nullable field the bound query is byte-identical to before. Four construction sites, all tests (quack, report, dir-sim). + +## Gates + +Independent row-at-a-time Kleene oracle over the ORIGINAL filter; NULL rows carry payloads that WOULD match (`5`, `0`, `7`). The required cases (`=`, `<>`, `NOT`, `AND`, `OR`, their negations, `[NOT] BETWEEN`, `[NOT] IN`, `IS [NOT] NULL`) on all four validity combinations, plus 300 random trees of depth ≤ 3, through both `lower` and `lower_fused`, rows and `COUNT(*)`. Pins: `NULL ≠ 0` (i32), `NULL ≠ ''` / `false` (u32 ordinal 0); the aggregate conventions (`COUNT(*)` vs `COUNT(x)`, SUM/MIN/MAX/AVG over valid only, empty ≠ zero via COUNT/MIN/AVG). Falsifier: `NOT (valid(x) ∧ x = 5)` and the raw unrewritten filter are both shown wrong on the fixture. The rewrite is one pass (each node visited once; the first cut rescanned every subtree, quadratic down a `NOT` chain — codex P2). Disable runs, each red: `Range` exempted from gating; F(leaf) without validity; NOT as gate-then-negate; F(AND) without De Morgan; T(leaf) without validity; `Semijoin` treated as an UNKNOWN leaf; `Draft::bind` without `sql_where`. + +Nullable fk (`fk::semijoin_over_a_null_fk_is_false_and_eq_via_is_unknown`): every third row NULL in the fk, payload `r % 4` pointing into a 4-row foreign table whose kept plane `{0,2}` and value lane `[9,1,9,1]` make NULL payloads match (≥ 10 trap rows asserted). Nine composites against an independent oracle (EXISTS = FALSE, via = UNKNOWN), both lowerings; can-fire both ways. + +## SQL READ coverage after this change (re-evaluated; nothing implemented) + +| item | class | priority | +|---|---|---| +| ⊘ ~~ordered `< <= > >=` on u32/u64 … **P0**~~ — downgraded, see the classification below | Quack + mask-risc | **P1 (bounded)** | +| ordered / range predicate on a joined dimension through an fk | Quack + mask-risc semantic gap; no current workload asks for it | P1 | +| i64 / float / decimal lanes | Quack + mask-risc; no current workload asks for it | P1 | +| multi-column / mixed GROUP BY, `SELECT DISTINCT x, y` | binder concern (destination binding → existing `Local`) | **P0 for the stack, not Quack** | +| NOT IN with a NULL literal, `x = NULL` | binder (constant UNKNOWN/FALSE leaf) | P2 | +| COALESCE / NULLIF / nullable arithmetic / projection arithmetic | binder (derived lanes) + result layer | P1 | +| `Agg::All` means "every one of n_rows", not "every filtered row" | Quack semantic bug (contract) | P1 | +| scalar `SUM` over no rows = 0, not NULL | result/finalization (pair with COUNT) | P1 | +| general / grouped `COUNT(DISTINCT)` | Quack + binder (presence fold over a bound destination) | P1 | +| HAVING OR / NOT, aggregate vs aggregate, AVG; per-aggregate filters | Quack semantic gap | P1 | +| foreign-side NULL / dangling fk in `EqU32Via` | Quack semantic gap (and a DuckDB case) | P1 | +| ORDER BY aggregate / TOP-k; multi-key ORDER BY | result/finalization | P1 / P2 | +| LIMIT / OFFSET over rows (bounded first-N) | result/finalization; physical binding for ordered projections | P1 | +| INNER/OUTER JOIN result rows (joined values; NULL extension) | result/egress contract; FULL OUTER | P1; FULL P2 | +| strings: `=`/`IN` bound; ranges/`LIKE 'p%'` need a sorted versioned codebook + ordered u32; `%x%`/regex bound over the dictionary | binder | P1 / P2 | +| set operations over rows (`OR`/`AND`/`AND NOT`); over values via a shared K | covered / result-layer K-combine | — / P1 | +| subqueries: EXISTS/IN via fk covered; scalar & correlated need Query→Query composition | Quack composition contract | P1 | +| window functions | Quack semantic gap (not started) | P2 | +| aliases, SQL parsing | frontend syntax | outside | +| INSERT / UPDATE / DELETE / MERGE / DDL | future DO | outside | + +⊘ The line that stood here ("P0 SQL READ gaps remain: ordered comparisons beyond i32") is withdrawn: a missing primitive is not a P0 without a workload that needs it. + +### Unsigned SQL-visible fields, classified (2026-10-05) + +| class | meaning | ordering | current fields | +|---|---|---|---| +| **A** numeric / order-semantic (year, sequence, timestamp) | `<` means something | needed | SAP `WORK_DAY` — an **i32** day lane, ranged with `GeI32`/`LeI32` (`lance-graph-sap/src/query.rs:35-36`); IAM prefix depth — `GeI32` (`dir-sim/src/lib.rs:141`); report measures / derived buckets — i32 (`report/src/exec.rs:242-255`). **None is u32/u64.** | +| **B** identity / ordinal / codebook / fk | equality only; `<` would invent semantics | must NOT exist | SAP `EMPLOYEE` NUMC (`EqU32`); IAM key / value ids (`EqU32`); report categorical ordinals — refused as `ReportError::OrderedCompareOnOrdinal` (`report/src/selection.rs:231`); `gremlin_parity` refuses `OrderedU32`. | +| **C** ordered address / codebook domain | order exists in the address space, not the value | `OrderedLaneWitness` + `Range` / binding | report row ranges (`Cmp::Range`, `selection.rs:179`). | + +**Verdict:** no current Report / IAM / SAP workload needs class-A u32/u64 ordering. Ordered unsigned comparison is **bounded P1**, opened only by a named class-A unsigned field; it does not block Destination Binding. The u64 kernel in ndarray existing is not a reason to wire it. + +**P0 SQL READ gaps after #1334:** none demonstrated. The NULL Boolean algebra is solved, and so is its binder wiring for `Draft` (nullability is a `BoundField` fact). A frontend that builds `Filter` directly, bypassing `Draft::bind`, still has to call `sql_where` itself. + +## OPEN + +- ⊘ ~~Whether `Binder` / `FieldKind` should carry nullability~~ — CLOSED in this PR (`BoundField::validity`). +- SAP's `CatsBatch::bind` carries NULL in OPTIONAL fields as in-lane SENTINELS, not validity planes (`lance-graph-sap/src/bind.rs`: dictionary code `0` :150, optional `pernr_d` `u32::MAX` :110, timestamp `0` :129) — a second NULL representation beside the one ruled here. Its only query (`query.rs:33-37`) reads required fields (`employee_number`, `work_date_utc`), so no wrong answer exists today; `=` on a sentinel-coded field is also safe (codes start at 1). The first SAP predicate over an optional field — or any `<>` / `NOT` on one — must first move that field to a validity plane and `sql_where`, else `<>` keeps NULL rows. P1, tied to that first query. +- Report's `Selection` builds filters without `Draft`; its plane 0 is table liveness, and no report field is declared nullable today. +- Ordered-compare substrate, VERIFIED-IN-CODE, for when a class-A field appears: ndarray HAS `gt/lt/ge/le_u64_to_mask` (`simd_masking_ops.rs:3570-3658`), unwired in mask-risc `Pred` and Quack `Cmp`; ndarray has NO ordered `u32` mask primitive. P1, not started. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 19a6c9d4f..38d584179 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -218 entries, 2026-08-06 .. 2026-10-05. +219 entries, 2026-08-06 .. 2026-10-05. | date | entry id | finding | file | |---|---|---|---| @@ -33,6 +33,7 @@ exactly one of them, never both. | 2026-10-05 | `report-pair-key-and-stack-recon` | | [2026-10-05-report-pair-key-and-stack-recon.md](2026-10-05-report-pair-key-and-stack-recon.md) | | 2026-10-05 | `quack-two-world-frontend` | | [2026-10-05-quack-two-world-frontend.md](2026-10-05-quack-two-world-frontend.md) | | 2026-10-05 | `quack-storage-portability` | | [2026-10-05-quack-storage-portability.md](2026-10-05-quack-storage-portability.md) | +| 2026-10-05 | `quack-sql-null-3vl` | | [2026-10-05-quack-sql-null-3vl.md](2026-10-05-quack-sql-null-3vl.md) | | 2026-10-05 | `fold-contract-and-keyed-sink-merge` | | [2026-10-05-fold-contract-and-keyed-sink-merge.md](2026-10-05-fold-contract-and-keyed-sink-merge.md) | | 2026-10-04 | `D-LXC-22` | | [2026-10-04-wordnet-clam-chaoda-scope-and-grammar-read-params.md](2026-10-04-wordnet-clam-chaoda-scope-and-grammar-read-params.md) | | 2026-10-04 | `D-HPS-1` | | [2026-10-04-spog-slab-hotplug-resolution.md](2026-10-04-spog-slab-hotplug-resolution.md) | diff --git a/crates/lance-graph-dir-sim/tests/quack_bind.rs b/crates/lance-graph-dir-sim/tests/quack_bind.rs index 85f9bde90..31ef9693b 100644 --- a/crates/lance-graph-dir-sim/tests/quack_bind.rs +++ b/crates/lance-graph-dir-sim/tests/quack_bind.rs @@ -28,6 +28,8 @@ impl Binder for IamBinder<'_> { Some(BoundField { col, kind: FieldKind::Code, + // IAM's dictionary ids are never NULL here. + validity: None, }) } fn code(&self, _: TableId, col: Col, literal: &str) -> Option { diff --git a/crates/lance-graph-quack/src/bind.rs b/crates/lance-graph-quack/src/bind.rs index a05f89580..91e160a23 100644 --- a/crates/lance-graph-quack/src/bind.rs +++ b/crates/lance-graph-quack/src/bind.rs @@ -79,6 +79,12 @@ pub struct BoundField { pub col: Col, /// What it compares against. pub kind: FieldKind, + /// The field's resident validity plane when it is NULLABLE, `None` when + /// every live row has a value. NULL is that plane's clear bit; the value + /// lane's payload there is never read as an answer. The binder owns this + /// because it owns the schema: a frontend cannot know it, and + /// [`Draft::bind`] needs it to make the bound `WHERE` three-valued. + pub validity: Option, } /// A consumer's resolution membrane: catalog plus codebook. Called only by @@ -270,6 +276,12 @@ impl Draft { /// [`Query`]. Calls the binder once per table, field and textual /// literal; the result holds none of them. /// + /// The bound filter is SQL three-valued over every nullable field a + /// predicate reads ([`BoundField::validity`]): it is passed through + /// [`Filter::sql_where`], so a NULL row is never kept by `=` or `<>`. A + /// query over non-nullable fields binds to exactly the filter it bound to + /// before. + /// /// # Errors /// /// The first [`BindError`]; nothing is minted or registered on failure. @@ -278,6 +290,7 @@ impl Draft { .table(&self.table) .ok_or_else(|| BindError::UnknownTable(self.table.clone()))?; let mut parts = vec![Filter::plane(b.live(t))]; + let mut nullable = Vec::new(); for p in &self.preds { if let Some(owner) = &p.table { if owner != &self.table { @@ -287,19 +300,28 @@ impl Draft { }); } } - parts.push(self.bind_pred(b, t, p)?); + let (leaf, f) = self.bind_pred(b, t, p)?; + if let Some(valid) = f.validity { + nullable.push((f.col, valid)); + } + parts.push(leaf); } let agg = match self.want { Want::Count => Agg::Count, Want::Rows => Agg::Rows, }; Ok(Query { - filter: Filter::and(parts), + filter: Filter::and(parts).sql_where(&nullable), agg, }) } - fn bind_pred(&self, b: &dyn Binder, t: TableId, p: &Predicate) -> Result { + fn bind_pred( + &self, + b: &dyn Binder, + t: TableId, + p: &Predicate, + ) -> Result<(Filter, BoundField), BindError> { let f = b .field(t, &p.field) .ok_or_else(|| BindError::UnknownField { @@ -331,7 +353,7 @@ impl Draft { } _ => return Err(mismatch()), }; - Ok(Filter::cmp(f.col, cmp)) + Ok((Filter::cmp(f.col, cmp), f)) } } @@ -452,10 +474,18 @@ mod tests { "smtp" => Some(BoundField { col: Col(0), kind: FieldKind::Code, + validity: None, }), "age" => Some(BoundField { col: Col(1), kind: FieldKind::I32, + validity: None, + }), + // The same lane, declared nullable: plane 1 is its validity. + "nage" => Some(BoundField { + col: Col(1), + kind: FieldKind::I32, + validity: Some(Mask(1)), }), _ => None, } @@ -470,6 +500,8 @@ mod tests { } /// 200 rows: smtp code = i % 3, age = 20 + i % 50, row 13 not live. + /// Plane 1 is `nage`'s validity: rows `i % 7 == 0` are NULL there, and + /// keep their age payload (so row 63 is NULL with payload 33). fn run(q: &Query) -> (Value, Vec) { let n = 200; let codes: Vec = (0..n).map(|i| (i % 3) as u32).collect(); @@ -477,8 +509,12 @@ mod tests { let mut live = vec![u64::MAX; words_for(n)]; live[3] &= (1u64 << (n % 64)) - 1; live[0] &= !(1 << 13); + let mut valid = vec![0u64; words_for(n)]; + for i in (0..n).filter(|i| !i.is_multiple_of(7)) { + valid[i / 64] |= 1 << (i % 64); + } let lanes = [LaneRef::U32(&codes), LaneRef::I32(&ages)]; - let masks: [&[u64]; 1] = [&live]; + let masks: [&[u64]; 2] = [&live, &valid]; let planes = Planes { n_rows: n, masks: &masks, @@ -548,6 +584,37 @@ mod tests { assert_eq!(run(&row13).1, vec![63, 113, 163]); } + /// A nullable field binds three-valued: a NULL row is kept by neither + /// `=` nor `<>`, even when its stale payload would match. Without the + /// binder's validity, `<>` keeps every NULL row whose payload differs. + #[test] + fn a_nullable_field_binds_three_valued() { + let users = Users::new(); + let live = |i: usize| i != 13; + let valid = |i: usize| !i.is_multiple_of(7); + let age = |i: usize| 20 + (i % 50) as i32; + let eq = table("users").where_eq("nage", 33).bind(&users).unwrap(); + // Row 63 has payload 33 but is NULL: SQL drops it. + assert_eq!(run(&eq).1, vec![113, 163]); + let ne = table("users") + .where_(FieldRef::new("users", "nage").ne(33)) + .bind(&users) + .unwrap(); + let want: Vec = (0..200) + .filter(|&i| live(i) && valid(i) && age(i) != 33) + .collect(); + assert_eq!(run(&ne).1, want); + // Can-fire: the same predicate on the non-nullable twin of the lane + // reads the payload of NULL rows and keeps them. + let raw = table("users") + .where_(FieldRef::new("users", "age").ne(33)) + .bind(&users) + .unwrap(); + let raw_rows = run(&raw).1; + assert!(raw_rows.iter().any(|&i| !valid(i))); + assert_ne!(raw_rows, want); + } + #[test] fn binding_fails_in_the_developers_vocabulary_and_never_mints() { let users = Users::new(); diff --git a/crates/lance-graph-quack/src/lib.rs b/crates/lance-graph-quack/src/lib.rs index c99e48a17..30064ec32 100644 --- a/crates/lance-graph-quack/src/lib.rs +++ b/crates/lance-graph-quack/src/lib.rs @@ -15,7 +15,7 @@ //! | a row iterator / `next()` volcano loop | the unit is a mask over `n_rows`, never a row | //! | a per-operator kernel library | every operator is a `MaskOp` composition; a new operator is a new LOWERING, never a new kernel | //! | a physical-plan `dyn Operator` chain | a plan is a `Program` — one flat op list, one terminal | -//! | a validity bitmap beside the data | the table's validity IS a resident mask plane ([`Filter::Plane`]); there is no separate NULL | +//! | a validity bitmap beside the data | the table's validity IS a resident mask plane ([`Filter::Plane`]); there is no separate NULL. A column's NULLs are ITS validity plane, and [`Filter::sql_where`] makes `WHERE` over them SQL three-valued | //! | a hash table for GROUP BY | a group is a mask; K groups are K gated equalities over the kept filter ([`lower_group_by`]) | //! //! The rule that keeps it honest: **this crate may build a [`Program`] and @@ -157,8 +157,12 @@ //! [`Agg::GroupSumI32`] for a key on THIS table, [`Agg::GroupSumViaI32`] for //! a key resolved through a foreign table's [`ForeignLane`] //! (`SUM(l.amount) GROUP BY p.country`) — the indirect-key group-sum is no -//! longer a gap. Absent, and everything the IR itself excludes: `ORDER BY`, -//! strings, three-valued NULL. +//! longer a gap. `WHERE` over nullable columns is SQL three-valued through +//! [`Filter::sql_where`]: each column's resident validity plane is the NULL +//! carrier, and UNKNOWN survives `NOT`/`AND`/`OR` up to the `WHERE`. Absent, +//! and everything the IR itself excludes: `ORDER BY`, strings, a NULL literal +//! and NULL-aware value expressions (`COALESCE`, `NULLIF`, nullable +//! arithmetic). #![forbid(unsafe_code)] @@ -456,6 +460,159 @@ impl Filter { ) } + /// `x IS NOT NULL`, where `valid` is `x`'s resident validity plane. + /// + /// The validity plane IS the NULL carrier: a row is NULL in `x` exactly + /// when its bit in `valid` is clear, whatever its value lane holds there. + /// This test is never UNKNOWN, so it is an ordinary plane leaf. + pub fn is_not_null(valid: Mask) -> Self { + Filter::Plane(valid) + } + + /// `x IS NULL`, where `valid` is `x`'s resident validity plane. Never + /// UNKNOWN. A table whose rows can be deleted still conjoins its liveness + /// plane as for any other filter: a deleted row has no `x` at all. + pub fn is_null(valid: Mask) -> Self { + Filter::Not(Box::new(Filter::Plane(valid))) + } + + /// This filter read as a SQL `WHERE` clause over nullable columns, + /// returned as an ordinary two-valued filter that keeps exactly the rows + /// for which SQL three-valued logic yields TRUE. + /// + /// `nullable` names each nullable column together with its resident + /// validity plane. Nothing else carries NULL: the value lane keeps + /// whatever payload it has at a NULL row (`0`, garbage, a stale value), + /// and that payload never takes part in the answer. `value 0, valid 1` is + /// a real zero; `value 0, valid 0` is NULL. There is no NULL value, no + /// tagged representation and no second bitmap. + /// + /// # Why a rewrite, and why both polarities + /// + /// Plain lowering is two-valued: [`Filter::Not`] complements its operand. + /// Gating a comparison on its validity plane is not enough on its own: + /// `NOT (valid(x) AND x = 5)` holds on every NULL row, but SQL's + /// `NOT (x = 5)` is UNKNOWN there and `WHERE` rejects it. UNKNOWN must + /// survive composition up to the `WHERE`. + /// + /// Each subformula `φ` is therefore lowered to one of two masks: + /// `T(φ)`, the rows where `φ` is TRUE, or `F(φ)`, the rows where it is + /// FALSE. A row in neither is UNKNOWN, so `known = T ∪ F`, and the result + /// is `T(self)`. The laws are the SQL ones: + /// + /// | `φ` | `T(φ)` | `F(φ)` | + /// |---|---|---| + /// | leaf on nullable `x` | `valid(x) ∧ leaf` | `valid(x) ∧ ¬leaf` | + /// | `Semijoin` on nullable `fk` | `valid(fk) ∧ leaf` | `¬(valid(fk) ∧ leaf)` | + /// | leaf on no nullable column, or a plane | `leaf` | `¬leaf` | + /// | `NOT ψ` | `F(ψ)` | `T(ψ)` | + /// | `ψ₁ AND … AND ψₙ` | `⋀ T(ψᵢ)` | `⋁ F(ψᵢ)` | + /// | `ψ₁ OR … OR ψₙ` | `⋁ T(ψᵢ)` | `⋀ F(ψᵢ)` | + /// + /// Each node is visited once, and `NOT` only flips polarity, so the + /// rewrite is linear in the size of `self`. The + /// output is built from the same `Cmp` / `Plane` / `And` / `Or` / `Not` + /// leaves the input uses, so [`lower`] and [`lower_fused`] run it with no + /// new op, and a `valid(x)` conjunct is an ordinary resident-plane gate + /// that the survivor skip uses. + /// + /// # Which leaves are nullable + /// + /// - [`Filter::Cmp`]: nullable when its column is listed. That includes + /// [`Cmp::Range`]: the executor reads only the row ordinal, but the + /// range stands for a comparison on its provenance lane, so a NULL in + /// that lane is UNKNOWN there too and the range is gated on its + /// validity. + /// - [`Filter::EqU32Via`]: nullable when its `fk` is listed. It is the + /// comparison `f.v = c` through the fk, and a NULL fk reaches no foreign + /// row, so the comparison is UNKNOWN there: `T = valid(fk) ∧ leaf`, + /// `F = valid(fk) ∧ ¬leaf`. + /// - [`Filter::Semijoin`]: NOT three-valued. It is + /// `EXISTS(SELECT 1 FROM foreign f WHERE f.rid = this.fk AND …)`, and with + /// a NULL fk the inner `f.rid = NULL` matches no row, so `EXISTS` is + /// FALSE — known, never UNKNOWN. Its `Gather` would still read the + /// foreign plane at the NULL row's stale payload, so a listed `fk` is + /// gated without becoming UNKNOWN: `T = valid(fk) ∧ leaf`, + /// `F = ¬(valid(fk) ∧ leaf)`. + /// - A NULL in the FOREIGN value lane is not modelled here. + /// - [`Filter::Plane`]: never nullable. A plane is a known Boolean, which + /// is what makes [`Filter::is_null`] / [`Filter::is_not_null`] exact. + /// + /// A filter that reads no listed column comes back unchanged, so a + /// non-nullable query lowers to exactly the program it lowered to + /// before. A subtree that reads no listed column is reused as written. + /// + /// Out of scope here: a NULL literal (`x = NULL`, `x NOT IN (1, NULL)`), + /// `COALESCE`/`NULLIF`, and nullable arithmetic. Aggregates are + /// unchanged: `COUNT(x)`, `SUM(x)` and the rest still take `valid(x)` in + /// their filter, as before. + pub fn sql_where(&self, nullable: &[(Col, Mask)]) -> Filter { + self.rewrite(nullable, true).unwrap_or_else(|| self.clone()) + } + + /// `Some(T(self))` when `want_true`, else `Some(F(self))`; `None` when + /// this subtree reads no nullable column, so it is two-valued and the + /// caller reuses it as written. Each node is visited once, so the + /// rewrite is linear even down a long `NOT` chain. See + /// [`Filter::sql_where`]. + fn rewrite(&self, nullable: &[(Col, Mask)], want_true: bool) -> Option { + match self { + Filter::Not(inner) => inner.rewrite(nullable, !want_true), + Filter::And(parts) | Filter::Or(parts) => { + let mapped: Vec> = parts + .iter() + .map(|p| p.rewrite(nullable, want_true)) + .collect(); + if mapped.iter().all(Option::is_none) { + return None; + } + let mapped = mapped + .into_iter() + .zip(parts) + .map(|(m, p)| m.unwrap_or_else(|| p.polar(want_true))) + .collect(); + // `T` keeps the connective and `F` takes its De Morgan dual. + Some(match (self, want_true) { + (Filter::And(_), true) | (Filter::Or(_), false) => Filter::And(mapped), + _ => Filter::Or(mapped), + }) + } + leaf => { + let valid = leaf.leaf_validity(nullable)?; + // `EXISTS` over a NULL fk is FALSE, not UNKNOWN: gate the + // leaf, then complement the gated leaf for `F`. + if let Filter::Semijoin { .. } = leaf { + let gated = Filter::And(vec![Filter::Plane(valid), leaf.clone()]); + return Some(gated.polar(want_true)); + } + Some(Filter::And(vec![ + Filter::Plane(valid), + leaf.polar(want_true), + ])) + } + } + } + + /// `self` when `want_true`, else `NOT self` — a two-valued subtree's `T`/`F`. + fn polar(&self, want_true: bool) -> Filter { + if want_true { + self.clone() + } else { + Filter::Not(Box::new(self.clone())) + } + } + + /// The validity plane of the nullable column this LEAF reads, if any. + fn leaf_validity(&self, nullable: &[(Col, Mask)]) -> Option { + let col = match *self { + Filter::Plane(_) => return None, + Filter::Cmp(col, _) => col, + Filter::EqU32Via { fk, .. } | Filter::Semijoin { fk, .. } => fk, + Filter::And(_) | Filter::Or(_) | Filter::Not(_) => return None, + }; + nullable.iter().find(|(c, _)| *c == col).map(|&(_, m)| m) + } + /// An `AND` whose children are ordered by how much of the gate each one /// KILLS — the V3 answer to DuckDB's `AdaptiveFilter`, and deliberately /// not DuckDB's algorithm. diff --git a/crates/lance-graph-quack/tests/sql_null_3vl.rs b/crates/lance-graph-quack/tests/sql_null_3vl.rs new file mode 100644 index 000000000..e6cd2db7e --- /dev/null +++ b/crates/lance-graph-quack/tests/sql_null_3vl.rs @@ -0,0 +1,830 @@ +//! SQL three-valued `WHERE` over nullable columns, against an independent +//! row-at-a-time 3VL oracle. +//! +//! The model under test: a nullable column is an ordinary value lane PLUS its +//! resident validity plane. A row is NULL in `x` exactly when its validity bit +//! is clear; the value lane's payload there is irrelevant. To make that +//! visible, the fixture puts payloads at NULL rows that WOULD match the +//! predicates (`5`, `0`, `7`). Any implementation that reads the payload of a +//! NULL row gets caught. +//! +//! The oracle never calls `sql_where`. It walks the ORIGINAL filter with +//! Kleene logic (TRUE / FALSE / UNKNOWN) and keeps the rows whose verdict is +//! TRUE. Every lowered program — `lower` and `lower_fused` — must keep the same +//! rows, and its `COUNT(*)` must agree. + +use lance_graph_mask_risc::{ + execute, execute_into, materialize_rows, words_for, Foreign, LaneRef, Out, Planes, Scratch, + Value, +}; +use lance_graph_quack::{ + avg_finish, lower, lower_avg, lower_fused, Agg, Cmp, Col, Filter, Mask, Query, +}; + +const X: Col = Col(0); +const Y: Col = Col(1); +const VX: Mask = Mask(0); +const VY: Mask = Mask(1); +const NULLABLE: [(Col, Mask); 2] = [(X, VX), (Y, VY)]; + +/// `None` is NULL. The payload stored at a NULL row is a separate field. +#[derive(Clone, Copy)] +struct Cell { + v: Option, + payload_if_null: i32, +} + +struct Table { + x: Vec, + y: Vec, + vx: Vec, + vy: Vec, + cells: Vec<(Cell, Cell)>, +} + +fn bit(w: &[u64], r: usize) -> bool { + w[r / 64] >> (r % 64) & 1 == 1 +} + +/// Every combination of `x ∈ {NULL, NULL, 0, 1, 5, 7, 10, 11}` and +/// `y ∈ {NULL, 7, 3}`. The two NULL x cells carry payloads `5` and `0`, the +/// NULL y cell carries `7`: each is the value some predicate below tests for. +/// The product (24 rows) is repeated with a stride so the table spans three +/// words and the tiles see both edge and interior words. +fn table() -> Table { + let xs = [ + Cell { + v: None, + payload_if_null: 5, + }, + Cell { + v: None, + payload_if_null: 0, + }, + Cell { + v: Some(0), + payload_if_null: 0, + }, + Cell { + v: Some(1), + payload_if_null: 0, + }, + Cell { + v: Some(5), + payload_if_null: 0, + }, + Cell { + v: Some(7), + payload_if_null: 0, + }, + Cell { + v: Some(10), + payload_if_null: 0, + }, + Cell { + v: Some(11), + payload_if_null: 0, + }, + ]; + let ys = [ + Cell { + v: None, + payload_if_null: 7, + }, + Cell { + v: Some(7), + payload_if_null: 0, + }, + Cell { + v: Some(3), + payload_if_null: 0, + }, + ]; + let mut cells = Vec::new(); + for rep in 0..7 { + for i in 0..xs.len() { + for j in 0..ys.len() { + // Rotate the product per repetition so row order is not the + // same pattern repeated (row bits land in different word slots). + cells.push((xs[(i + rep) % xs.len()], ys[(j + 2 * rep) % ys.len()])); + } + } + } + let n = cells.len(); + let mut t = Table { + x: Vec::with_capacity(n), + y: Vec::with_capacity(n), + vx: vec![0; words_for(n)], + vy: vec![0; words_for(n)], + cells, + }; + for (r, (cx, cy)) in t.cells.iter().enumerate() { + t.x.push(cx.v.unwrap_or(cx.payload_if_null)); + t.y.push(cy.v.unwrap_or(cy.payload_if_null)); + if cx.v.is_some() { + t.vx[r / 64] |= 1 << (r % 64); + } + if cy.v.is_some() { + t.vy[r / 64] |= 1 << (r % 64); + } + } + t +} + +// ── The oracle: Kleene logic over the ORIGINAL filter ───────────────────── + +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +enum Tv { + T, + F, + U, +} + +fn tv(b: bool) -> Tv { + if b { + Tv::T + } else { + Tv::F + } +} + +fn eval(f: &Filter, t: &Table, r: usize) -> Tv { + match f { + Filter::Cmp(col, cmp) => { + let cell = if *col == X { + t.cells[r].0 + } else { + t.cells[r].1 + }; + let Some(v) = cell.v else { return Tv::U }; + tv(match *cmp { + Cmp::EqI32(c) => v == c, + Cmp::NeI32(c) => v != c, + Cmp::LtI32(c) => v < c, + Cmp::LeI32(c) => v <= c, + Cmp::GtI32(c) => v > c, + Cmp::GeI32(c) => v >= c, + // A range bound on the ordered lane: membership by row + // ordinal, UNKNOWN (above) where that lane is NULL. + Cmp::Range { lo, hi } => (lo as usize..hi as usize).contains(&r), + other => panic!("oracle: unsupported {other:?}"), + }) + } + Filter::Plane(m) => { + let w = if *m == VX { &t.vx } else { &t.vy }; + tv(bit(w, r)) + } + Filter::Not(inner) => match eval(inner, t, r) { + Tv::T => Tv::F, + Tv::F => Tv::T, + Tv::U => Tv::U, + }, + Filter::And(parts) => { + let vs: Vec = parts.iter().map(|p| eval(p, t, r)).collect(); + if vs.contains(&Tv::F) { + Tv::F + } else if vs.contains(&Tv::U) { + Tv::U + } else { + Tv::T + } + } + Filter::Or(parts) => { + let vs: Vec = parts.iter().map(|p| eval(p, t, r)).collect(); + if vs.contains(&Tv::T) { + Tv::T + } else if vs.contains(&Tv::U) { + Tv::U + } else { + Tv::F + } + } + other => panic!("oracle: unsupported {other:?}"), + } +} + +fn oracle(f: &Filter, t: &Table) -> Vec { + (0..t.cells.len()) + .filter(|&r| eval(f, t, r) == Tv::T) + .collect() +} + +// ── Running a lowered program ──────────────────────────────────────────── + +fn planes(t: &Table) -> ([&[u64]; 2], [LaneRef<'_>; 2]) { + ([&t.vx, &t.vy], [LaneRef::I32(&t.x), LaneRef::I32(&t.y)]) +} + +/// The rows a filter keeps, run through `lower` (`fused == false`) or +/// `lower_fused`, with `COUNT(*)` checked against the kept set. +fn kept(f: &Filter, t: &Table, fused: bool) -> Vec { + let n = t.cells.len(); + let (masks, lanes) = planes(t); + let pl = Planes { + n_rows: n, + masks: &masks, + lanes: &lanes, + }; + let lower_fn = if fused { lower_fused } else { lower }; + let p = lower_fn(&Query { + filter: f.clone(), + agg: Agg::Rows, + }) + .expect("lowers"); + let mut s = Scratch::for_program(&p, n).expect("carves"); + let mut out = vec![0u64; words_for(n)]; + execute_into(&p, &pl, &Foreign::NONE, &mut s, Out::Mask(&mut out)).expect("runs"); + let rows = materialize_rows(&out, n); + + let c = lower_fn(&Query { + filter: f.clone(), + agg: Agg::Count, + }) + .expect("lowers"); + let mut s = Scratch::for_program(&c, n).expect("carves"); + assert_eq!( + execute(&c, &pl, &mut s, None).expect("runs"), + Value::Count(rows.len()), + "COUNT(*) disagrees with the kept rows" + ); + rows +} + +fn check(name: &str, f: &Filter, t: &Table) { + let want = oracle(f, t); + let w = f.sql_where(&NULLABLE); + for fused in [false, true] { + assert_eq!(kept(&w, t, fused), want, "{name}: fused={fused}"); + } +} + +fn x(c: Cmp) -> Filter { + Filter::cmp(X, c) +} +fn y(c: Cmp) -> Filter { + Filter::cmp(Y, c) +} +fn not(f: Filter) -> Filter { + Filter::negate(f) +} + +/// The required cases. Each is checked on every (x valid/NULL) × (y +/// valid/NULL) combination, since the fixture is their full product. +#[test] +fn where_is_sql_three_valued_on_the_required_cases() { + let t = table(); + let cases: Vec<(&str, Filter)> = vec![ + ("x = 5", x(Cmp::EqI32(5))), + ("x <> 5", x(Cmp::NeI32(5))), + ("NOT (x = 5)", not(x(Cmp::EqI32(5)))), + ( + "x = 5 AND y = 7", + Filter::and([x(Cmp::EqI32(5)), y(Cmp::EqI32(7))]), + ), + ( + "x = 5 OR y = 7", + Filter::or([x(Cmp::EqI32(5)), y(Cmp::EqI32(7))]), + ), + ( + "NOT (x = 5 AND y = 7)", + not(Filter::and([x(Cmp::EqI32(5)), y(Cmp::EqI32(7))])), + ), + ( + "NOT (x = 5 OR y = 7)", + not(Filter::or([x(Cmp::EqI32(5)), y(Cmp::EqI32(7))])), + ), + ( + "x BETWEEN 1 AND 10", + Filter::and([x(Cmp::GeI32(1)), x(Cmp::LeI32(10))]), + ), + ( + "x NOT BETWEEN 1 AND 10", + not(Filter::and([x(Cmp::GeI32(1)), x(Cmp::LeI32(10))])), + ), + ("x IN (1,2,3)", Filter::in_i32(X, [1, 2, 3])), + ("x NOT IN (1,2,3)", not(Filter::in_i32(X, [1, 2, 3]))), + ("x IS NULL", Filter::is_null(VX)), + ("x IS NOT NULL", Filter::is_not_null(VX)), + ("NOT (x IS NULL)", not(Filter::is_null(VX))), + ( + "x IS NULL OR x = 5", + Filter::or([Filter::is_null(VX), x(Cmp::EqI32(5))]), + ), + ("NOT NOT (x = 5)", not(not(x(Cmp::EqI32(5))))), + ]; + for (name, f) in &cases { + check(name, f, &t); + } + + // Anti-vacuity: the fixture must reach every Kleene outcome of the + // composite cases, and each of the four validity combinations. + let both = Filter::and([x(Cmp::EqI32(5)), y(Cmp::EqI32(7))]); + let outcomes: Vec = (0..t.cells.len()).map(|r| eval(&both, &t, r)).collect(); + for o in [Tv::T, Tv::F, Tv::U] { + assert!(outcomes.contains(&o), "fixture never yields {o:?}"); + } + for (vx, vy) in [(true, true), (false, true), (true, false), (false, false)] { + assert!( + t.cells + .iter() + .any(|(a, b)| a.v.is_some() == vx && b.v.is_some() == vy), + "fixture lacks x valid={vx}, y valid={vy}" + ); + } +} + +/// FAILS IF: `NOT (x = 5)` is lowered as `NOT (valid(x) AND x = 5)`. +/// +/// That form is the obvious one, and it is wrong: it holds on every NULL row, +/// where SQL's verdict is UNKNOWN and `WHERE` rejects the row. The raw, +/// unrewritten filter is wrong too, differently: it reads the payload of the +/// NULL rows. Both are shown to disagree with the oracle on this fixture, so +/// the oracle is not vacuous against them. +#[test] +fn gating_a_leaf_before_negation_is_not_three_valued() { + let t = table(); + let f = not(x(Cmp::EqI32(5))); + let want = oracle(&f, &t); + let naive = not(Filter::and([Filter::is_not_null(VX), x(Cmp::EqI32(5))])); + let nulls: Vec = (0..t.cells.len()) + .filter(|&r| t.cells[r].0.v.is_none()) + .collect(); + for fused in [false, true] { + let got = kept(&naive, &t, fused); + assert_ne!(got, want, "naive gating must be wrong (fused={fused})"); + assert!( + nulls.iter().all(|r| got.contains(r)), + "the naive form keeps every NULL row" + ); + let raw = kept(&f, &t, fused); + assert_ne!(raw, want, "the unrewritten filter reads NULL payloads"); + assert_eq!(kept(&f.sql_where(&NULLABLE), &t, fused), want); + } + assert!( + nulls.iter().all(|r| !want.contains(r)), + "WHERE rejects UNKNOWN" + ); +} + +/// FAILS IF: NULL is treated as a value. `x = 0` keeps the rows whose x is a +/// real zero and none of the NULL rows whose payload is `0`; `x <> 0` keeps +/// neither kind of NULL. +#[test] +fn null_is_not_zero() { + let t = table(); + let eq0 = x(Cmp::EqI32(0)).sql_where(&NULLABLE); + let got = kept(&eq0, &t, false); + assert!(!got.is_empty()); + assert!(got.iter().all(|&r| t.cells[r].0.v == Some(0))); + let null_zero_payload: Vec = (0..t.cells.len()) + .filter(|&r| t.cells[r].0.v.is_none() && t.x[r] == 0) + .collect(); + assert!( + !null_zero_payload.is_empty(), + "fixture needs a NULL with payload 0" + ); + let ne0 = kept(&x(Cmp::NeI32(0)).sql_where(&NULLABLE), &t, false); + assert!(null_zero_payload + .iter() + .all(|r| !got.contains(r) && !ne0.contains(r))); +} + +fn lcg(s: &mut u64) -> u64 { + *s = s + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + *s >> 33 +} + +fn random_filter(s: &mut u64, depth: u32) -> Filter { + let leaf = |s: &mut u64| -> Filter { + let col = if lcg(s).is_multiple_of(2) { X } else { Y }; + let v = [0, 3, 5, 7, 10][(lcg(s) % 5) as usize]; + match lcg(s) % 8 { + 0 => Filter::cmp(col, Cmp::EqI32(v)), + 1 => Filter::cmp(col, Cmp::NeI32(v)), + 2 => Filter::cmp(col, Cmp::LtI32(v)), + 3 => Filter::cmp(col, Cmp::GeI32(v)), + 4 => Filter::cmp(col, Cmp::GtI32(v)), + 5 => Filter::cmp(col, Cmp::LeI32(v)), + 6 => Filter::is_null(if col == X { VX } else { VY }), + _ => Filter::is_not_null(if col == X { VX } else { VY }), + } + }; + if depth == 0 || lcg(s).is_multiple_of(4) { + return leaf(s); + } + match lcg(s) % 3 { + 0 => Filter::negate(random_filter(s, depth - 1)), + 1 => Filter::and((0..2 + lcg(s) % 2).map(|_| random_filter(s, depth - 1))), + _ => Filter::or((0..2 + lcg(s) % 2).map(|_| random_filter(s, depth - 1))), + } +} + +/// FAILS IF: any composition law is wrong — De Morgan duals, double +/// negation, or a leaf's validity on either polarity — over 300 random trees +/// of depth ≤ 3 mixing nullable comparisons, IS [NOT] NULL and NOT/AND/OR, +/// lowered both ways. +#[test] +fn random_trees_agree_with_the_oracle_both_lowerings() { + let t = table(); + let mut s = 0x3_7a1_u64; + let mut differs_from_raw = 0; + for i in 0..300 { + let f = random_filter(&mut s, 3); + check(&format!("random #{i}: {f:?}"), &f, &t); + if kept(&f, &t, false) != oracle(&f, &t) { + differs_from_raw += 1; + } + } + // Anti-vacuity: the raw two-valued lowering must be wrong on a real share + // of these trees, or the corpus does not exercise NULL at all. + assert!( + differs_from_raw > 30, + "only {differs_from_raw} trees exercised NULL" + ); +} + +/// FAILS IF: a filter that reads no nullable column is rewritten. The +/// non-nullable fast path must lower to the identical program, both ways. +#[test] +fn non_nullable_filters_are_returned_unchanged() { + let mut s = 0xfa57_u64; + for _ in 0..50 { + let f = random_filter(&mut s, 3); + assert_eq!(f.sql_where(&[]), f); + for fused in [false, true] { + let lower_fn = if fused { lower_fused } else { lower }; + let q = |filter| Query { + filter, + agg: Agg::Count, + }; + assert_eq!( + lower_fn(&q(f.sql_where(&[]))).ok(), + lower_fn(&q(f.clone())).ok(), + "fused={fused}" + ); + } + } + // Only y nullable: a subtree over x alone is reused as written. + let f = Filter::and([x(Cmp::EqI32(5)), not(y(Cmp::EqI32(7)))]); + let w = f.sql_where(&[(Y, VY)]); + match &w { + Filter::And(parts) => assert_eq!(parts[0], x(Cmp::EqI32(5))), + other => panic!("expected an AND, got {other:?}"), + } +} + +/// FAILS IF: the aggregate conventions regress. `COUNT(*)` counts rows; +/// `COUNT(x)`, `SUM(x)`, `MIN(x)`, `MAX(x)` and `AVG(x)` see valid `x` only +/// (validity in their filter, as before this change); and an empty set of +/// contributions is distinguishable from a real zero. +#[test] +fn aggregates_keep_their_null_conventions() { + let t = table(); + let n = t.cells.len(); + let (masks, lanes) = planes(&t); + let pl = Planes { + n_rows: n, + masks: &masks, + lanes: &lanes, + }; + let run = |filter: Filter, agg: Agg| -> Value { + let p = lower(&Query { filter, agg }).expect("lowers"); + let mut s = Scratch::for_program(&p, n).expect("carves"); + execute(&p, &pl, &mut s, None).expect("runs") + }; + let w = y(Cmp::EqI32(7)).sql_where(&NULLABLE); + let rows = oracle(&y(Cmp::EqI32(7)), &t); + let xs: Vec = rows.iter().filter_map(|&r| t.cells[r].0.v).collect(); + assert!(xs.len() < rows.len(), "some selected rows must have NULL x"); + + let with_x = Filter::and([w.clone(), Filter::is_not_null(VX)]); + assert_eq!( + run(w.clone(), Agg::Count), + Value::Count(rows.len()), + "COUNT(*)" + ); + assert_eq!( + run(with_x.clone(), Agg::Count), + Value::Count(xs.len()), + "COUNT(x)" + ); + assert_eq!( + run(with_x.clone(), Agg::SumI32(X)), + Value::SumI64(xs.iter().map(|&v| i64::from(v)).sum()), + "SUM(x)" + ); + assert_eq!( + run(with_x.clone(), Agg::MinI32(X)), + Value::OptI32(xs.iter().copied().min()) + ); + assert_eq!( + run(with_x.clone(), Agg::MaxI32(X)), + Value::OptI32(xs.iter().copied().max()) + ); + + let plan = lower_avg(&with_x, X).expect("lowers"); + let exec = |p: &lance_graph_mask_risc::Program| { + let mut s = Scratch::for_program(p, n).expect("carves"); + execute(p, &pl, &mut s, None).expect("runs") + }; + let (Value::SumI64(sum), Value::Count(cnt)) = (exec(&plan.sum), exec(&plan.count)) else { + panic!("AVG plan shapes") + }; + assert_eq!( + avg_finish(sum, cnt as u64), + Some(xs.iter().map(|&v| f64::from(v)).sum::() / xs.len() as f64) + ); + + // No contribution is not a real zero: the only-NULL-x group has COUNT(x) + // = 0, MIN(x) = None and AVG(x) = None, while a valid x = 0 group has + // COUNT(x) = 1 and MIN(x) = Some(0). + let only_null = Filter::and([Filter::is_null(VX), Filter::is_not_null(VX)]); + assert_eq!(run(only_null.clone(), Agg::Count), Value::Count(0)); + assert_eq!(run(only_null.clone(), Agg::MinI32(X)), Value::OptI32(None)); + let plan = lower_avg(&only_null, X).expect("lowers"); + let (Value::SumI64(sum), Value::Count(cnt)) = (exec(&plan.sum), exec(&plan.count)) else { + panic!("AVG plan shapes") + }; + assert_eq!(avg_finish(sum, cnt as u64), None); + let zero = x(Cmp::EqI32(0)).sql_where(&NULLABLE); + assert!(matches!(run(zero.clone(), Agg::Count), Value::Count(c) if c > 0)); + assert_eq!(run(zero, Agg::MinI32(X)), Value::OptI32(Some(0))); +} + +/// FAILS IF: NULL is folded into the ordinal of `''` or into `false` on a +/// `u32` code lane. Both are ordinary values that happen to be encoded as +/// ordinal `0` (an empty string's codebook ordinal, a boolean's `false`); +/// NULL is the cleared validity bit, whatever ordinal the payload holds. +/// The same law as for `i32`, on the lane type text and booleans bind to. +#[test] +fn null_is_not_the_empty_string_or_false() { + let n = 130; + const Z: Col = Col(0); + const VZ: Mask = Mask(0); + // Every third row is NULL with payload 0; the rest alternate 0 / 1. + let z: Vec = (0..n) + .map(|r| if r % 3 == 0 { 0 } else { (r % 2) as u32 }) + .collect(); + let mut vz = vec![0u64; words_for(n)]; + for r in (0..n).filter(|r| r % 3 != 0) { + vz[r / 64] |= 1 << (r % 64); + } + let masks: [&[u64]; 1] = [&vz]; + let lanes = [LaneRef::U32(&z)]; + let pl = Planes { + n_rows: n, + masks: &masks, + lanes: &lanes, + }; + let rows = |f: Filter| -> Vec { + let p = lower(&Query { + filter: f.sql_where(&[(Z, VZ)]), + agg: Agg::Rows, + }) + .expect("lowers"); + let mut s = Scratch::for_program(&p, n).expect("carves"); + let mut out = vec![0u64; words_for(n)]; + execute_into(&p, &pl, &Foreign::NONE, &mut s, Out::Mask(&mut out)).expect("runs"); + materialize_rows(&out, n) + }; + let eq0 = rows(Filter::cmp(Z, Cmp::EqU32(0))); + let ne0 = rows(Filter::cmp(Z, Cmp::NeU32(0))); + let null = rows(Filter::is_null(VZ)); + let want_eq0: Vec = (0..n).filter(|&r| r % 3 != 0 && r % 2 == 0).collect(); + let want_ne0: Vec = (0..n).filter(|&r| r % 3 != 0 && r % 2 == 1).collect(); + let want_null: Vec = (0..n).filter(|&r| r % 3 == 0).collect(); + assert_eq!(eq0, want_eq0, "'' / false: real ordinal-0 rows only"); + assert_eq!(ne0, want_ne0, "a NULL row is neither = 0 nor <> 0"); + assert_eq!(null, want_null); + assert_eq!( + eq0.len() + ne0.len() + null.len(), + n, + "TRUE / FALSE / NULL partition the rows" + ); +} + +/// A `Cmp::Range` bound on a nullable lane is UNKNOWN where that lane is +/// NULL, even though the executor reads only the row ordinal: the range +/// stands for a comparison on its provenance lane. Can-fire: the raw range +/// keeps NULL rows inside `[lo, hi)`. +#[test] +fn a_range_on_a_nullable_lane_is_gated() { + let t = table(); + let range = x(Cmp::Range { lo: 10, hi: 100 }); + check("range", &range, &t); + check("NOT range", ¬(range.clone()), &t); + check( + "range OR y = 7", + &Filter::or([range.clone(), y(Cmp::EqI32(7))]), + &t, + ); + let raw = kept(&range, &t, false); + assert!(raw.iter().any(|&r| t.cells[r].0.v.is_none())); + assert_ne!(raw, oracle(&range, &t)); +} + +/// A deep alternating `NOT` / `AND` / `OR` chain with nullable leaves at +/// every level still agrees with the oracle through both lowerings, and the +/// same chain over non-nullable leaves only comes back unchanged. Depth 96 +/// keeps every polarity flip, connective, and De Morgan dual in play without +/// testing the debug stack. +#[test] +fn a_deep_mixed_chain_is_rewritten_correctly_and_a_non_nullable_one_is_untouched() { + let t = table(); + let leaves = [ + x(Cmp::EqI32(5)), + y(Cmp::NeI32(7)), + x(Cmp::GtI32(0)), + Filter::is_null(VY), + ]; + let mut f = x(Cmp::LtI32(10)); + for d in 0..96 { + let leaf = leaves[d % leaves.len()].clone(); + f = match d % 3 { + 0 => not(Filter::and([f, leaf])), + 1 => Filter::or([not(f), leaf]), + _ => not(not(Filter::and([leaf, f]))), + }; + } + check("deep mixed chain", &f, &t); + assert_ne!(f.sql_where(&NULLABLE), f, "the chain reads nullable leaves"); + // Anti-vacuity: the raw chain is wrong on this fixture. + assert_ne!(kept(&f, &t, false), oracle(&f, &t)); + + // The same shape over leaves that read nothing nullable is returned as is. + let mut g = Filter::cmp(Col(9), Cmp::EqI32(1)); + for d in 0..96 { + let leaf = Filter::cmp(Col(9), Cmp::NeI32(d)); + g = if d % 2 == 0 { + not(Filter::and([g, leaf])) + } else { + Filter::or([not(g), leaf]) + }; + } + assert_eq!(g.sql_where(&NULLABLE), g); +} + +// ── Nullable foreign key: Semijoin is FALSE, EqU32Via is UNKNOWN ────────── + +mod fk { + use lance_graph_mask_risc::{ + execute_into, materialize_rows, words_for, Foreign, ForeignPlane as FPlane, LaneRef, Out, + Planes, Scratch, + }; + use lance_graph_quack::{ + lower, lower_fused, Agg, Col, Filter, ForeignLane, ForeignPlane, Mask, Query, + }; + + const FK: Col = Col(0); + const VFK: Mask = Mask(0); + /// The foreign table: 4 rows, kept plane `{0, 2}`, value lane `[9, 1, 9, 1]`. + const KEPT: [u64; 1] = [0b0101]; + const FVAL: [u32; 4] = [9, 1, 9, 1]; + const N: usize = 130; + + /// Every row's fk payload is `r % 4`, so NULL rows (every third row) point + /// at live, MATCHING foreign rows as often as valid rows do. + fn table() -> (Vec, Vec) { + let mut fk = Vec::with_capacity(N); + let mut valid = vec![0u64; words_for(N)]; + for r in 0..N { + fk.push((r % 4) as u32); + if !r.is_multiple_of(3) { + valid[r / 64] |= 1 << (r % 64); + } + } + (fk, valid) + } + + #[derive(Clone, Copy, PartialEq, Eq, Debug)] + enum Tv { + T, + F, + U, + } + + /// Independent SQL semantics, row at a time. + fn eval(f: &Filter, fk: &[u32], r: usize) -> Tv { + let is_null = r.is_multiple_of(3); + let b = |x: bool| if x { Tv::T } else { Tv::F }; + match f { + // EXISTS(… WHERE f.rid = NULL …) is FALSE. + Filter::Semijoin { .. } if is_null => Tv::F, + Filter::Semijoin { .. } => b(KEPT[0] >> fk[r] & 1 == 1), + // f.v = 9 through a NULL fk is UNKNOWN. + Filter::EqU32Via { .. } if is_null => Tv::U, + Filter::EqU32Via { v, .. } => b(FVAL[fk[r] as usize] == *v), + Filter::Not(i) => match eval(i, fk, r) { + Tv::T => Tv::F, + Tv::F => Tv::T, + Tv::U => Tv::U, + }, + Filter::And(ps) => ps.iter().fold(Tv::T, |a, p| match (a, eval(p, fk, r)) { + (Tv::F, _) | (_, Tv::F) => Tv::F, + (Tv::U, _) | (_, Tv::U) => Tv::U, + _ => Tv::T, + }), + Filter::Or(ps) => ps.iter().fold(Tv::F, |a, p| match (a, eval(p, fk, r)) { + (Tv::T, _) | (_, Tv::T) => Tv::T, + (Tv::U, _) | (_, Tv::U) => Tv::U, + _ => Tv::F, + }), + other => panic!("oracle: unsupported {other:?}"), + } + } + + fn kept(f: &Filter, fused: bool) -> Vec { + let (fk, valid) = table(); + let masks = [&valid[..]]; + let lanes = [LaneRef::U32(&fk)]; + let pl = Planes { + n_rows: N, + masks: &masks, + lanes: &lanes, + }; + let fplanes = [FPlane { + words: &KEPT, + rows: 4, + }]; + let flanes = [LaneRef::U32(&FVAL)]; + let foreign = Foreign { + planes: &fplanes, + lanes: &flanes, + }; + let lower_fn = if fused { lower_fused } else { lower }; + let p = lower_fn(&Query { + filter: f.clone(), + agg: Agg::Rows, + }) + .expect("lowers"); + let mut s = Scratch::for_program(&p, N).expect("carves"); + let mut out = vec![0u64; words_for(N)]; + execute_into(&p, &pl, &foreign, &mut s, Out::Mask(&mut out)).expect("runs"); + materialize_rows(&out, N) + } + + fn sj() -> Filter { + Filter::semijoin(FK, ForeignPlane(0)) + } + fn via() -> Filter { + Filter::eq_u32_via(FK, ForeignLane(0), 9) + } + fn not(f: Filter) -> Filter { + Filter::negate(f) + } + + /// The nullable-fk falsifier: a NULL fk whose stale payload points at a + /// kept, matching foreign row. `Semijoin` must answer FALSE there (so its + /// negation KEEPS the row) and `EqU32Via` UNKNOWN (so neither it nor its + /// negation keeps it). Treating `Semijoin` as UNKNOWN drops the row from + /// `NOT semijoin`; reading the payload keeps it in `semijoin`. + #[test] + fn semijoin_over_a_null_fk_is_false_and_eq_via_is_unknown() { + let (fk, _) = table(); + let nullable = [(FK, VFK)]; + // Anti-vacuity: NULL rows whose payload points at a kept, matching row. + let traps = (0..N) + .filter(|&r| r.is_multiple_of(3) && KEPT[0] >> fk[r] & 1 == 1) + .count(); + assert!(traps >= 10, "fixture lost its traps ({traps})"); + + let cases = [ + ("semijoin", sj()), + ("NOT semijoin", not(sj())), + ("eq_via", via()), + ("NOT eq_via", not(via())), + ("semijoin OR NOT semijoin", Filter::or([sj(), not(sj())])), + ("eq_via OR NOT eq_via", Filter::or([via(), not(via())])), + ("NOT (semijoin AND eq_via)", not(Filter::and([sj(), via()]))), + ( + "NOT semijoin AND NOT eq_via", + Filter::and([not(sj()), not(via())]), + ), + ( + "NOT (NOT semijoin OR eq_via)", + not(Filter::or([not(sj()), via()])), + ), + ]; + for (name, f) in &cases { + let want: Vec = (0..N).filter(|&r| eval(f, &fk, r) == Tv::T).collect(); + let w = f.sql_where(&nullable); + for fused in [false, true] { + assert_eq!(kept(&w, fused), want, "{name}: fused={fused}"); + } + } + + // Can-fire, both directions: NULL rows ARE kept by `NOT semijoin` (it + // is FALSE there, a known answer), and are NOT kept by `NOT eq_via`. + let nulls: Vec = (0..N).filter(|r| r.is_multiple_of(3)).collect(); + let not_sj = kept(¬(sj()).sql_where(&nullable), false); + assert!(nulls.iter().all(|r| not_sj.contains(r))); + let not_via = kept(¬(via()).sql_where(&nullable), false); + assert!(nulls.iter().all(|r| !not_via.contains(r))); + // And the unrewritten semijoin reads the stale payload: it is wrong. + let raw = kept(&sj(), false); + assert!(nulls.iter().any(|r| raw.contains(r))); + assert_ne!(raw, kept(&sj().sql_where(&nullable), false)); + } +} diff --git a/crates/lance-graph-report/tests/quack_bind.rs b/crates/lance-graph-report/tests/quack_bind.rs index 9723b573b..bcbd2f727 100644 --- a/crates/lance-graph-report/tests/quack_bind.rs +++ b/crates/lance-graph-report/tests/quack_bind.rs @@ -34,6 +34,8 @@ impl Binder for ReportBinder<'_> { Some(BoundField { col: Col(id.0 as u16), kind, + // The fixture's fields carry no NULLs. + validity: None, }) } fn code(&self, _: TableId, col: Col, literal: &str) -> Option {