diff --git a/.claude/board/entries/2026-10-05-report-pair-key-and-stack-recon.md b/.claude/board/entries/2026-10-05-report-pair-key-and-stack-recon.md new file mode 100644 index 000000000..f69f3f113 --- /dev/null +++ b/.claude/board/entries/2026-10-05-report-pair-key-and-stack-recon.md @@ -0,0 +1,77 @@ +# Report Pair experiment superseded; destination-binding gap and stack recon (2026-10-05) + +**Status:** ⊘ SUPERSEDED IN PART (2026-10-05, same PR, before merge) — the Pair execution path described under "The cut" was REVERTED; this PR now ships only the corrected gap doc in `exec.rs` and this record. See "Correction" below. Recon findings and decisions 2–5 stand. ~~MEASURED — the code change and its tests are in this PR.~~ The recon findings are VERIFIED-IN-CODE against z8run `3a8a758`, lance-graph `97a3610d`, OGAR `e5de84e`, rs-graph-llm `e824977` and rig `d165343`. It ratifies the five decisions below and nothing more. + +## Correction (2026-10-05) — why the cut below was superseded + +`GroupAddr::Pair` addresses `hi · stride + lo`: the **dense product** of two dimension domains. Lowering every two-ordinal report to it made the Cartesian product the execution geometry because the primitive existed, not because any consumer demanded a dense cube. The intended fold model is the PowerShell-Hashtable / Excel-Pivot one: a row resolves its accumulator **destination** directly, and accumulator state scales with the **demanded / resolved destination universe**, not with an accidental Cartesian product. For IAM-sized dimensions (64k × 64k) a Pair sink is 4·10⁹ slots whatever the data demands. + +- **What stays true:** Pair was useful *evidence* that Report re-implemented what Quack can fold, and the old doc's "substrate gap" claim was stale. +- **What was wrong:** Pair is not the general Pivot/Join execution model. Destination binding/resolution + the EXISTING keyed fold is. +- **What the source shows (VERIFIED-IN-CODE):** + - ndarray/mask-risc/Quack have ONE keyed-reduction walker with three addresses — `Lane`, `Via` (`table[fk[i]]`), `Pair` — all writing a caller-sized K-slot sink (ndarray `simd_masking_ops.rs` `masked_group_*`, mask-risc `ir.rs` `GroupKey`, Quack `GroupAddr`, lgj `plan_lower.rs` Local/Via). **That keyed fold IS the execution primitive once a destination is bound:** a compact resident destination ordinal folds through `GroupAddr::Local`; `fk[row]` resolves to a foreign destination ordinal through `GroupAddr::Via`; K is the destination count. **The missing capability is destination binding / coordinate resolution for Report:** `CoordSpec` has only `Field`, `Bucket` and `MaskSet`, so it cannot express a functional-reference coordinate (`user → department`), a composed functional route, or several semantic coordinates bound to ONE compact destination ordinal (codex review, #1331). + - The only compact structure is Report's `Layout::Sparse` RESULT index, and reaching it costs (`exec.rs` sparse arm): one discovery population scan for each ORDINAL partition dimension (a domain-sized `GroupReduce Count`); then, with T = the product of the partition radices — DISCOVERED members for an ordinal dimension, the FULL configured radix for a `Bucket` / `MaskSet` dimension — one population fold for every tuple in T for every `FoldState` (`for_each_tuple` × states). Main fold cost: **T × FoldStates** population scans, plus the discovery scans, plus one selection-materialization scan when the `ReusedMask` carrier is chosen. E.g. 100 × 100 observed members × 4 FoldStates = 40,000 fold scans after discovery — not one scan per member (codex review, #1331). This is the partition-rescan architecture the destination-binding gap exists to eliminate. +- **Reverted:** `fold_major`, `fold_addr`, the Pair SUM normalization, the product budget, the Pair strides and decode, the Pair-pinning tests. Report is back to main's planner. +- **Kept:** `GroupAddr::Pair` in Quack/mask-risc/ndarray (correct for an explicitly dense mixed-radix problem; untouched here). +- **Shipped:** `exec.rs`'s gap doc now names the missing capability as destination binding / coordinate resolution for Report, with the existing `Local` / `Via` keyed fold acknowledged as the execution primitive. + +## The cut (⊘ SUPERSEDED — reverted before merge; kept as the record of what was tried) + +`lance-graph-report` planned two-dimensional reports as **one population fold per partition tuple per FoldState** (see the corrected cost under "Correction"). Its module doc called the composite-key group fold "a named substrate gap". That gap had already closed: Quack has `GroupAddr::Pair { hi, lo, stride }`. + +The physical planner now picks a **fold major**: the widest remaining ordinal whose product with the fold key's domain fits `domain_buffer_budget`. It folds both coordinates in ONE pass, through the pair key (`exec.rs` `fold_addr`). + +| case | before | after | +|---|---|---| +| 20×30 plan | 20 passes | 1 pass | +| `population_scans` | 80 | 4 | + +- Bucket coordinates, mask-set coordinates and a third ordinal stay partitions. Quack has no value-derived or multi-plane group key. +- Quack has no pair-keyed SUM terminal, so a pair-keyed SUM lowers to the NULL-preserving `GroupReduce { SumI32 }`. Its raw sink is normalized with Quack's own `normalize_group_sink` (empty → 0, the SUM identity). Count, Min and Max seed exactly the report's identities (0, `i64::MAX`, `i64::MIN`). Honest limit: the NULL-preserving sum is defined up to `GROUP_SUM_SYM_MAX_ROWS` = 2^32 − 1 rows, one fewer than `GroupSumI32`, so a pair-keyed SUM over a plane of exactly 2^32 rows is refused, not wrapped. Values (including `i32::MIN`) are unaffected. + +**Tests:** +- New `pair_key_folds_two_ordinals_in_one_pass_and_equals_per_member_passes`: pair ≡ per-member on every cell, total and hidden-axis merge, over a mostly-empty 600-cell space. + - Budget threshold: 600 pairs, 599 does not (can-fire / can-stay-silent). + - **Disable run:** with the normalization removed, it goes red (`agnostic.rs:564`), then green when restored. +- `t9` now pins the pair programs, plus a per-member twin under a tight budget. +- The pass-budget test moved to three ordinals so it still fires at 20. + +**Historical observation from review (not active work):** the Pair choice was greedy, not pass-minimal — with domains 100 × 60 × 60 and a 3600 budget it ran 3600 partition passes where folding 60 × 60 and partitioning over 100 would need 100. This is NOT an open optimization for Report: optimizing which dense product to fold is the superseded model. `GroupAddr::Pair` remains valid only where a consumer explicitly demands a dense mixed-radix destination universe; it is not the future implementation of sparse Pivot/Join/grouping. + +**Other runs:** +- z8run-lance (`pivot_flow`, handle size under 128 bytes, `z8run_plan_is_the_native_plan`, zero-refold pivot): 10/10. +- report-ogar: 4/4. +- Report suite: 33/33. +- clippy `-D warnings` clean. + +## Recon findings that set the scope + +- **`pivot_flow()` already runs on Quack:** `lance-execute` → `ReportPlan::execute` → `quack::lower` → mask-risc. ReportPlan is a Quack frontend that duplicates some Quack semantics. Remaining duplicates: IN as an OR of equalities (Quack has `Filter::in_u32`), MEAN/NULL finalize (Quack has `lower_avg` + `GroupAgg::SumI32` + normalize), and its own Selection/Scalar membrane. +- **Quack has no execute and no result handle.** It stops at `Query` → `lower` → `Program`. +- **z8run-lance is not wired into z8run.** It is a workspace exclude, and `register_lance_nodes` is never called. +- **z8run-core runs a JSON row engine:** `database` (up to 1,000 rows into `FlowMessage`) → `aggregator` (group-by over JSON); `batch`/`loop` stream a population as messages. +- **graph-flow infers Kanban lifecycle from workflow status** (`storage_kanban.rs:158-173`: `Completed → Commit`, `Error → Prune`, `Commit → CognitiveWork`). This bypasses `can_transition_to`. graph-flow-kanban also no longer matches the contract (`KanbanMove.libet_offset_us`, `GateDecision {reason}`). +- **Session status is never persisted**, and the engine never builds `ExecutionStatus::Error`. +- **No carrier holds (coordinate space × version):** mask-risc `Planes`, `AbiBatch` and Quack `Query` identify populations by length. The z8run Envelope and the report generation use a free `u32`. +- **No dependency edge** between z8run, graph-flow and ogar-loco in any direction. + +## DECISION (operator, 2026-10-05) — the provenance of these five rows is that choice; the evidence is the recon above + +1. ~~**First PR** = the two-ordinal partition → `GroupAddr::Pair`, one pass.~~ ⊘ SUPERSEDED (operator, same day): Pair is not the fold model; this PR is reduced to revert + corrected record. See "Correction". +2. **The numeric Quack `Query` is the canonical boundary.** `bind::Draft` and `ReportPlan` are SIBLING frontends; so are a SQL frontend, an IAM frontend, a z8run island compiler and a Rig tool frontend. `Draft` stays small. A maximal pure z8run island lowers to one bound Query/program, not to "one Draft". +3. **Retire the graph-flow → KanbanColumn inference.** Keep generic session replay and resume. graph-flow emits facts and results (task completed, waiting for input, result ref, action receipt); the canonical Kanban + Revision owns lifecycle. graph-flow is not wired to the cycle-seal driver either. +4. **Enforce coordinate space × version at the population execution and result boundaries** (execute(Program, World), ResultRef, workspace). Do not put the dataset version into the `Query` IR to satisfy the invariant. Binder provenance (version-dependent CAM ordinals) is a separate question. +5. **No `Quack::execute` and no `Quack::ResultHandle` yet.** First determine who owns the execution membrane between a numeric Program and a versioned World. It is probably a backend adapter above mask-risc (#1330: "Quack owns query semantics; storage supplies capabilities"). + +**REVISIT WHEN:** a second frontend needs a capability that only `Draft` growth could give; or the execution membrane's owner is settled. + +## OPEN + +- **The missing capability — destination binding / coordinate resolution for Report** (not a fold primitive): semantic coordinate(s) → BIND / RESOLVE DESTINATION → compact destination ordinal or functional route → the EXISTING `Local` / `Via` keyed fold. Report's `CoordSpec` (`Field`, `Bucket`, `MaskSet`) cannot yet express a functional-reference coordinate (`user → department`, although `GroupAddr::Via` / `Filter::EqU32Via` exist), a composed functional route, or several semantic coordinates bound to ONE compact destination ordinal. **Law:** a join/pivot used only to determine an aggregate destination must compile to destination binding/resolution + fold — not to an intermediate relation, and not automatically to a dense product; accumulator state scales with the demanded / resolved destination universe. The normal path binds beforehand (resident ordinal, CAM / codebook binding, `Via` or composed functional reference). A query-local compact tuple interner is only a possible fallback for a genuinely unbound destination universe, not the canonical path. No representation, type or variant is chosen here. Contract first — see the fold-contract recon. +- ~~**`CoordSpec` has no functional-reference coordinate**~~ — folded into the missing-capability item above (it is its first, separable case). +- Owner of the execution membrane (Program + World → ResultRef). It is needed by both z8run and graph-flow. +- Whether the merge-law re-roll of totals is a Quack fold or presentation. +- Whether `with_roles` may hide dimensions, since it merges over them: a GROUP BY done in the view. +- `ReportPlan::pivot`/`axis` docs say "pure metadata", but adding a coordinate changes the physical key (a data pivot). +- OGAR `handle_submit` runs `CapabilityExecutor` with no gate; the gate exists only in rs-graph-llm's `dispatch_via`. +- The `graph-flow-action-ogar` production path mints V1 (`NodeGuid::new`), reads the classid canon-low, and hardwires `guard_field_value = None`. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 663abe3ff..34c7272f8 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,11 +25,12 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -216 entries, 2026-08-06 .. 2026-10-05. +217 entries, 2026-08-06 .. 2026-10-05. | date | entry id | finding | file | |---|---|---|---| | 2026-10-05 | `text-to-numeric-boundary-inventory` | | [2026-10-05-text-to-numeric-boundary-inventory.md](2026-10-05-text-to-numeric-boundary-inventory.md) | +| 2026-10-05 | `report-pair-key-and-stack-recon` | | [2026-10-05-report-pair-key-and-stack-recon.md](2026-10-05-report-pair-key-and-stack-recon.md) | | 2026-10-05 | `quack-two-world-frontend` | | [2026-10-05-quack-two-world-frontend.md](2026-10-05-quack-two-world-frontend.md) | | 2026-10-05 | `quack-storage-portability` | | [2026-10-05-quack-storage-portability.md](2026-10-05-quack-storage-portability.md) | | 2026-10-04 | `D-LXC-22` | | [2026-10-04-wordnet-clam-chaoda-scope-and-grammar-read-params.md](2026-10-04-wordnet-clam-chaoda-scope-and-grammar-read-params.md) | diff --git a/crates/lance-graph-report/src/exec.rs b/crates/lance-graph-report/src/exec.rs index bf6fa9575..b7ff9a6a5 100644 --- a/crates/lance-graph-report/src/exec.rs +++ b/crates/lance-graph-report/src/exec.rs @@ -42,9 +42,31 @@ //! The honest limit: a partition costs one pass per member tuple, so a plan //! whose partition side is high-cardinality AND densely observed exceeds the //! pass budget and is REFUSED with [`ReportError::PassBudget`] rather than -//! run slowly or allocated densely. The primitive that would lift it — a -//! composite-key (multi-lane) group fold in `ndarray::simd` / mask-risc — is -//! a named substrate gap, not something to hand-roll here. +//! run slowly or allocated densely. The missing capability is **destination +//! binding / coordinate resolution for Report** — not a fold primitive. Once +//! a compact destination ordinal or a functional reference is bound, the +//! existing keyed fold IS the execution primitive: a resident destination +//! ordinal folds through `GroupAddr::Local`, and `fk[row]` resolves to a +//! foreign destination ordinal through `GroupAddr::Via`, each into a +//! caller-sized K-slot sink where K is the destination count. What Report +//! lacks is the binding step: `CoordSpec` has only `Field`, `Bucket` and +//! `MaskSet`, so it cannot express a functional-reference coordinate +//! (`user → department`), a composed functional route, or several semantic +//! coordinates bound to ONE compact destination ordinal — and so it +//! partitions instead. +//! +//! The law: a join or pivot used only to determine an aggregate destination +//! must compile to destination binding / resolution + fold — never to an +//! intermediate relation, and never automatically to a dense product — so +//! accumulator state scales with the demanded / resolved destination universe, +//! not with an accidental Cartesian product of the dimension domains. The +//! normal path binds destinations beforehand (a resident ordinal, a CAM / +//! codebook binding, a `Via` or composed functional reference). A query-local +//! compact interner is only a possible fallback for a destination universe +//! that genuinely cannot be bound beforehand, not the canonical path. (Quack's +//! `GroupAddr::Pair`, `hi · stride + lo`, addresses the dense product; it is +//! valid only for a consumer that explicitly demands a dense mixed-radix +//! destination universe, and this planner does not lower to it.) use std::collections::HashMap; use std::sync::Arc;