quack: SQL three-valued WHERE over nullable columns (value lane + validity plane)

[AdaWorldAPI]

Quack continues to use ordinary value lanes plus resident validity planes. This PR makes Boolean filtering over nullable values SQL-3VL correct. It does not introduce a general-purpose SQL NULL value representation.

value 0, valid 1 is a real zero. value 0, valid 0 is NULL. NULL never becomes a value: this PR adds no boxed or tagged NULL, no second bitmap and no expression interpreter.

The bug this fixes (P0 from the SQL coverage reconnaissance)

Filter::Not lowers to a plain complement. Over a nullable column that silently keeps NULL rows:

• NOT (x = 5), x <> 5 and x NOT IN (…) all keep rows where x is NULL.
• Even the obvious repair, NOT (valid(x) AND x = 5), is wrong: SQL's NOT (x = 5) is UNKNOWN on a NULL row, and WHERE rejects UNKNOWN.

The change

Filter::sql_where(&self, nullable: &[(Col, Mask)]) -> Filter, plus Filter::is_null / Filter::is_not_null over the validity plane.

Each subformula lowers to T(φ), the rows where it is TRUE, or F(φ), the rows where it is FALSE. A row in neither is UNKNOWN. WHERE keeps T(root).

φ	T(φ)	F(φ)
leaf on nullable x (Cmp including Range, EqU32Via by its fk)	valid(x) ∧ leaf	valid(x) ∧ ¬leaf
Semijoin on nullable fk	valid(fk) ∧ leaf	¬(valid(fk) ∧ leaf)
leaf on no nullable column, or a plane	leaf	¬leaf
NOT ψ	F(ψ)	T(ψ)
AND	⋀ T	⋁ F
OR	⋁ T	⋀ F

• mask-risc is unchanged. The output is built from the same Cmp/Plane/And/Or/Not nodes, so lower and lower_fused run it as is.
• The validity check is an ordinary gate. valid(x) lowers like any resident plane, so the existing survivor skip uses it.
• Non-nullable fast path: a filter or subtree that reads no listed column comes back unchanged and lowers to the identical program.
• One pass: the rewrite visits each node once. A subtree with no nullable column is reported as two-valued and reused as written.
• How each leaf is treated:
  • Cmp: nullable by its column.
  • Cmp::Range executes over the row ordinal but retains its provenance Col. If that semantic lane is nullable, the Range predicate is gated by that lane's validity plane, so NULL yields UNKNOWN and WHERE rejects it. Physical implementation detail ≠ semantic type.
  • EqU32Via: nullable by its fk. With a NULL fk, f.v = c is UNKNOWN.
  • Semijoin: not three-valued. It is EXISTS(SELECT 1 FROM foreign f WHERE f.rid = this.fk AND …), and with a NULL fk that is FALSE, a known answer. The fk is still gated, because Gather would otherwise read the stale payload at the NULL row. So NOT semijoin keeps NULL-fk rows, and NOT eq_via does not.
  • Plane: never nullable. A plane is a known Boolean, which is what makes IS [NOT] NULL exact.
• Binder wiring:
  • BoundField gains validity: Option<Mask>. The binder owns the schema, so it owns nullability.
  • Draft::bind passes the bound filter through sql_where, so a bound WHERE over a nullable field is three-valued without the caller doing anything.
  • Non-nullable binds are byte-identical to before.
  • All four construction sites are tests (quack, report, dir-sim).

Out of scope:

• COALESCE, NULLIF, nullable arithmetic and projection;
• a NULL literal (x = NULL, NOT IN (…, NULL));
• foreign-side NULL;
• outer-join null extension;
• NULLS FIRST/LAST.

Aggregates are unchanged: COUNT(x) / SUM / MIN / MAX / AVG still take valid(x) in their filter.

Gates

tests/sql_null_3vl.rs checks against an independent row-at-a-time Kleene oracle that evaluates the original filter. Rows where a value is NULL carry payloads (5, 0, 7) that would match the predicates, so any read of a NULL payload shows up.

• Required cases: =, <>, NOT (x = 5), AND, OR, NOT (… AND …), NOT (… OR …), [NOT] BETWEEN, [NOT] IN, IS [NOT] NULL.
  • Run on all four (x valid/NULL) × (y valid/NULL) combinations.
  • Also run on 300 random trees of depth ≤ 3.
  • Both lower and lower_fused; both kept rows and COUNT(*).
• Nullable Range: a_range_on_a_nullable_lane_is_gated. NULL rows lie physically inside the ordinal range [10, 100), and the raw range keeps them (can-fire). Range, NOT range and range OR y match the oracle.
• Deep chain: a_deep_mixed_chain_is_rewritten_correctly_and_a_non_nullable_one_is_untouched. A 96-deep alternating NOT/AND/OR chain matches the oracle through both lowerings, and the raw chain does not. The same shape over non-nullable leaves comes back unchanged.
• Nullable fk: fk::semijoin_over_a_null_fk_is_false_and_eq_via_is_unknown.
  • NULL-fk rows carry payloads that point at kept, matching foreign rows; at least 10 such trap rows are asserted.
  • Nine Semijoin / EqU32Via composites are checked against an oracle where EXISTS is FALSE and the via comparison is UNKNOWN, through both lowerings.
  • Can-fire in both directions.
• Binder: bind::tests::a_nullable_field_binds_three_valued.
  • Row 63 is NULL with payload 33, and = 33 drops it.
  • <> 33 keeps no NULL row.
  • The non-nullable twin of the field does keep NULL rows (can-fire).
• Pins:
  • NULL ≠ 0 (i32);
  • NULL ≠ '' / false (u32 ordinal 0);
  • the aggregate conventions: COUNT(*) vs COUNT(x), valid-only SUM/MIN/MAX/AVG, and no contributions ≠ zero.
• Falsifier: NOT (valid(x) ∧ x = 5) and the raw unrewritten filter are both shown to disagree with the oracle on this fixture.
• Non-nullable filters: returned unchanged, with identical lowered programs from both lower and lower_fused.
• Disable runs. Each of these mutations turned the suite red:
  • F(leaf) without validity;
  • NOT as gate-then-negate;
  • F(AND) without De Morgan;
  • T(leaf) without validity;
  • Semijoin treated as an UNKNOWN leaf;
  • Draft::bind without sql_where;
  • Range exempted from gating.
• Checks:
  • the lance-graph-quack, lance-graph-report and dir-sim quack_bind suites pass;
  • fmt and clippy -D warnings are clean.
• Rustdoc: the broken-link errors are pre-existing and fail identically on main.

Board

entries/2026-10-05-quack-sql-null-3vl.md records the decision and the re-evaluated SQL READ coverage.

• Unsigned fields, classified:

  • A, numeric: SAP work day, IAM depth and report measures. All are already i32.
  • B, identity / ordinal / fk: equality only.
  • C, ordered address: Range.

  No current Report / IAM / SAP workload needs class-A u32/u64 ordering, so ordered unsigned comparison is bounded P1.

• No P0 SQL READ gap is demonstrated.

• P1: SAP's optional fields carry NULL as in-lane sentinels. Its only query reads required fields, so it is correct today; the first predicate over an optional field must move that field to a validity plane.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCEP2fZdHYdMCtcEVcTpS2

---

Generated by Claude Code

---

Generated by Claude Code

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Essentials
• Run ID: 8ede5a4c-d157-4c6b-8711-0134e2b6514f

📥 Commits

Reviewing files that changed from the base of the PR and between f46cd60 and 43e96a4.

📒 Files selected for processing (6)

• .claude/board/entries/2026-10-05-quack-sql-null-3vl.md
• crates/lance-graph-dir-sim/tests/quack_bind.rs
• crates/lance-graph-quack/src/bind.rs
• crates/lance-graph-quack/src/lib.rs
• crates/lance-graph-quack/tests/sql_null_3vl.rs
• crates/lance-graph-report/tests/quack_bind.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• .claude/board/entries/2026-10-05-quack-sql-null-3vl.md

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

---

📝 Walkthrough

Walkthrough

The change adds Filter::is_null, Filter::is_not_null, and Filter::sql_where for nullable columns. Binding supplies validity masks to the filter rewrite. Tests compare SQL three-valued WHERE results through both lowering paths, and a decision record documents behavior and coverage.

Changes

Nullable SQL filtering

Layer / File(s)	Summary
NULL checks and SQL WHERE rewrite crates/lance-graph-quack/src/lib.rs	Adds NULL-check constructors and a sql_where rewrite. Nullable comparisons and EqU32Via use validity masks; a semijoin on a NULL foreign key remains FALSE. Filters without listed nullable columns remain unchanged.
Binder validity and predicate wiring crates/lance-graph-quack/src/bind.rs, crates/lance-graph-dir-sim/tests/quack_bind.rs, crates/lance-graph-report/tests/quack_bind.rs	Adds optional validity to BoundField. Draft::bind collects predicate validity masks and applies sql_where. The IAM and Report fixtures set validity to None.
Oracle and nullable predicate coverage crates/lance-graph-quack/tests/sql_null_3vl.rs	Adds an independent Kleene-logic oracle and tests for comparisons, Boolean filters, ranges, membership, NULL checks, randomized filter trees, aggregates, u32 lanes, and nullable foreign keys. Tests exercise both lowering paths.
SQL READ coverage record .claude/board/entries/2026-10-05-quack-sql-null-3vl.md, .claude/board/entries/README.md	Adds a decision record and index entry. The record describes the rewrite, test coverage, SQL READ classifications, and remaining open items.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Suggested reviewers: claude

Merge Risk: ⚪ Minimal · up to 43e96

This change makes WHERE filters over nullable columns keep only rows that evaluate to TRUE, so NULL rows are excluded. Filters that do not touch nullable columns behave as before. No concrete defect was identified, and the change appears ready to merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 54.35% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 46 functions across 5 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: SQL three-valued WHERE behavior for nullable columns. It is specific and related to the pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 54.35% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 46 functions across 5 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{I’m a rabbit who hops through the NULLs in the lane,
TRUE rows come through; UNKNOWN does not remain.
Validity masks mark where the values reside,
And NOT, AND, and OR keep their truth on each side.
Two lowering paths meet the oracle’s view,
Then I nibble a carrot and call the tests true.}

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 0218ca64-5e55-473d-ab82-a31f9856e58b)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f46cd60645

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".