diff --git a/.claude/blackboard.md b/.claude/blackboard.md index 0b82496c..1950ed11 100644 --- a/.claude/blackboard.md +++ b/.claude/blackboard.md @@ -1,3 +1,21 @@ +## 2026-09-17 (19) — G8 named: a tree-depth column (`lzcnt(bswap(x)) >> 2`) is the missing primitive for basin-local ranking; popcount is only its tie-break + +Filed, not built. Full text in `masking-ops-state.md` § OUTLOOK G8 and the +2026-09-17 EPIPHANIES entry. Two things for the next session: + +- **Loose end:** the gate is a named consumer call site that ranks by depth. + Candidates: lance-graph `FacetCascade` tail ranking (basin-local similarity) + and `NiblePath::common_prefix_depth` (the packed-`u64` carrier where this + `lzcnt` IS the fold — `ISS-NIBLEPATH-FOLD-IS-CARRIER-2-UNMASKED`). Either + one, once it exists as a call, licenses the build. +- **Decision recorded:** do NOT add a `popcount` variant for this; the existing + `popcount_batch_u64` already serves the tie-break. The gap is `lzcnt` (+ + `bswap`), and (8) on the lance-graph board had already named the `lzcnt` + half. + +Consistent with (18): stripping HEEL/HIP makes the operand start power-of-two, +so the tail-padding question never arises for this op. + ## 2026-09-16 (18) — the tail OPTIMISATION IS INERT ON EVERY POWER-OF-TWO POPULATION >= 512 ROWS, including the 4096-row tile Established before writing the rewrite (15)-(17) argued for, and it changes diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 7e2a8d2b..ecb41724 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,5 +1,38 @@ # ndarray — Epiphanies (append-only) +## 2026-09-17 — On a tree-path register the metric is lzcnt, popcount is the tie-break, and little-endian bytes need one bswap first +**Status:** FINDING (the layout and the instruction facts) + NAMED GAP (G8, not built) +**Scope:** @simd-savant @family-codec-smith domain:masking-ops domain:v3-facet +**Cross-ref:** `.claude/knowledge/masking-ops-state.md` § OUTLOOK G8; lance-graph +`E-THREE-CARRIERS-THREE-FOLDS-1`, `ISS-NIBLEPATH-FOLD-IS-CARRIER-2-UNMASKED`, +LATEST_STATE 2026-09-15 (6)–(8); blackboard (18) + +Within a basin every V3 facet shares HEEL and HIP by construction, so the +6-tier LCP saturates and the information is in the tail — bytes 8..16, tiers +2–5, one aligned `u64` at a compile-time offset. Loading it is a PEEK (an +address), not a mask (a reassembly), so register ops on it are legitimate +under the carrier doctrine; and stripping the shared prefix is a load-offset +choice that costs nothing. Two consequences the workspace had not written down: + +1. **Padding is useless once you can strip.** The tail descent / `[u64; 8]` + zero-padding story exists because a 128-bit facet operand is ragged in a + 512-bit register. A 64-bit tail is not: eight tails tile a zmm exactly. + Blackboard (18) found the tail optimisation inert on power-of-two + populations *because there is no tail*; this is the same fact from the + operand side — choose the width that starts power-of-two. +2. **The op on it is `lzcnt`, not `popcount`.** Each tier byte is a 4-level + 4-ary centroid tree, so the tail is a path and its metric is depth of + divergence. Popcount is position-blind on a path (leaf flip == root flip). + Ranking = `lzcnt(bswap(a ^ b)) >> 2`; popcount belongs only to the + tie-break among equal-depth candidates (`x & below(depth)`) — coarse by + depth, fine by density. + +The `bswap` is load-bearing and is the trap: LE byte order puts the root tier +at the LOW byte, but nibble order inside a byte is MSB-coarse, so neither +`tzcnt` nor `lzcnt` alone reads the path in one direction. G8 records the +falsifier that catches a `bswap`-less implementation. Gate before building is +the G5 rule: one named consumer call site. + ## 2026-07-29 — Which PACKAGE pulls a dep decides whether it can consume you **Status:** FINDING **Scope:** @simd-savant @truth-architect domain:build-graph diff --git a/.claude/knowledge/masking-ops-state.md b/.claude/knowledge/masking-ops-state.md index 7f7573c2..a6dd9f2e 100644 --- a/.claude/knowledge/masking-ops-state.md +++ b/.claude/knowledge/masking-ops-state.md @@ -200,6 +200,132 @@ unary-with-constant, and that narrowing is exactly the register in which the constant, record G7 as *deliberately absent* **in the IR's docs**, so a future session does not "fix" it. +**G8 — `lzcnt_bswap_u64_to_u8` (tree-depth column) — NAMED 2026-09-17, not +built.** The V3 facet's tail (bytes 8..16 = tiers 2–5, one aligned `u64` at a +compile-time offset — a PEEK, not a gather) is a **tree path**. State the +carving before the op, because "tier" and "byte" are not the same unit: the +V3 facet is `classid(4) | payload(12)`, and `RailSpec::v3_facet` +(`src/hpc/clam_v3.rs`) walks **six sequential `(u8:u8)` levels at key offsets +`4..16`, stride 2** — so a tier is TWO bytes, one per axis, not one byte. The +`u64` operand at `8..16` therefore covers tiers 2–5, i.e. **4 tiers × 2 bytes += 8 bytes = 16 nibbles**. Each *byte* within a tier +is a **16-ary nibble hierarchy** — OGAR canon's *"1 hex digit = 1 nibble = 1 +level of the 16-ary tree (`FAN_OUT=16`)"* — high nibble coarse. The metric on +a tree path is depth-of-divergence, which at that granularity on a `u64` is +`lzcnt(a ^ b) >> 2` — position-aware. + +**Which factorization of 256 this op commits to, stated because OGAR carries +two and they differ by 2×.** Beside the nibble reading above, the canon also +says *"256 = 4⁴ — each codebook is built as a 4-level 4-ary centroid +hierarchy"*. That one is the **Morton-interleaved** centroid space, where a +nibble of `FacetTier::morton()` is a 2 bit × 2 bit quad-tree level across BOTH +axes at once — a different tree, one level finer per step, over a different +operand. G8 reads the **raw** tail `u64`, not the interleaved code, so it is +the nibble tree and the shift is `>> 2` (depth 0..16 over 8 bytes). A 4-ary +depth over the raw bytes would be `>> 1` (0..32). Both are monotone in +leading-equal-bits, so **ranking is identical either way** — the granularity +only bites where depth is consumed as a *value*, which is exactly the in-cell +tie-break `x & below(depth)` proposed below. Do not cite 4⁴ as the +justification for `>> 2`; it justifies `>> 1`. (Corrected 2026-09-18 after a +sibling session measured the two against a fixture; the first draft of this +entry asserted 4⁴ while shipping the nibble shift.) `popcount(a ^ b)` is **position-blind** +on it (a leaf-nibble flip counts the same as a root-nibble flip; lance-graph +LATEST_STATE 2026-09-15 (8): *"popcount also finds elephant : Wal"*), so the +existing `popcount_batch_u64` / `xor_popcount` are the wrong primitive for +ranking and the right one only for the in-cell tie-break (`x & below(depth)`). + +**The byte-order wrinkle the first implementation will get backwards:** the +facet is little-endian (tier 2 at the LOW byte of the tail `u64`) but the +hierarchy inside a byte is MSB-coarse. `tzcnt` gets byte order right and +nibble order wrong; `lzcnt` the reverse. One `bswap` reconciles them: + +```text +depth_nibbles = lzcnt(bswap(a ^ b)) >> 2 // 0..16, stepless, no branch +``` + +That is three scalar instructions (`xor`, `bswap`, `lzcnt`) or, as a column, +`vpshufb` (byte reverse, **AVX-512BW** at zmm width) + `vplzcntq` +(**AVX-512CD**) over 8 rows per zmm — BOTH feature bits, not CD alone; a +target with CD but not BW needs a different byte reversal. Per 8-row group +there is **no operand padding and no tail descent** — eight tails tile a zmm +exactly, which is +the whole reason the `u64` width is the right one here (and consistent with +blackboard (18): the tail optimisation is inert on power-of-two populations +because there is no tail; stripping HEEL/HIP makes the operand *start* +power-of-two). **That is a claim about the OPERAND, not the population:** a +row count not divisible by 8 still leaves a column remainder, exactly as the +slice-level `U64x8` ops handle with `pad_tail`. No multiple-of-eight input +contract is stated or intended, so an implementation keeps a remainder path +and tests non-multiple-of-eight lengths. What is eliminated is the +*within-operand* padding a narrower width would need, not the last partial +group. Consumer: basin-local similarity in lance-graph (`FacetCascade` +tail) and the CAKES nearest-ranking on `NiblePath` (lance-graph +`ISS-NIBLEPATH-FOLD-IS-CARRIER-2-UNMASKED`, the packed-`u64` carrier where the +same `lzcnt` IS the fold). Realizations: `vplzcntq` on avx512cd; avx2 has no +vector lzcnt — polyfill via the float-exponent trick or a scalar peel (measure +which); NEON has `clz` on 32-bit lanes (two halves + select); wasm/scalar flat; +and the **nightly** `core::simd` arm (`src/simd_nightly/`) — six in total, +which is the count the falsifier below means. +Pre-registered falsifier: the column op must equal the scalar +`(a ^ b).swap_bytes().leading_zeros() >> 2` on every row of a 64k fixture at +all six realizations, AND a disable-run without the `bswap` must fail on a +fixture whose divergence is in a low nibble of a high tier (that is the +backwards-implementation trap, and the test must be able to see it). +**Plus a third arm, because the first two cannot catch a wrong shift:** both +compare the column op against `(a ^ b).swap_bytes().leading_zeros() >> 2`, +which is the same formula in a different spelling, so they agree even if the +shift is wrong. The third arm builds the fixture *independently* — plant a +divergence at a KNOWN tier and nibble, assert the KNOWN depth — and is the +only one that discriminates `>> 2` from `>> 1`. **Measured** (2026-09-18, all +16 positions, against a scalar model): for tier `t` ∈ 2..5, byte-in-tier `b` +(0 = `lo`, the LOWER address; 1 = `hi`) and nibble-in-byte `k` (0 = high/MSB, +1 = low/LSB), + +``` +depth = 4*(t-2) + 2*b + k // 0..15; 16 iff the tails are identical +``` + +— exact at every one of the 16 positions. (A first draft of this arm wrote +`2*(t-2) + n`, which is right only for tier 2 and drifts by `2*(t-2)` +thereafter; each tier is 2 bytes = **4** nibbles, not 2. It was caught by +running the model, which is the point of the arm.) + +**Caveat the measurement surfaced, and the reason the consumer gate matters +more than it looked.** `FacetTier` is `{ lo, hi }` with `lo` at the LOWER +address, so after the `bswap` a single `lzcnt` over the raw tail walks +`lo` BEFORE `hi` within every tier — i.e. it alternates between the two +chains every two nibbles. But `hi_chain` and `lo_chain` are documented as two +**orthogonal** hierarchies (`facet.rs`: *"the `hi` chain prefix-routes one +hierarchy, the `lo` chain the orthogonal one"*), not coarse and fine of one. +So raw-tail depth is a **mixed-axis** metric: still a valid monotone +tie-breaker, NOT a depth in either hierarchy. A consumer that wants one +axis needs a per-chain fold (the existing `hi_distance` / `lo_distance`) or a +deinterleave before the `lzcnt`. + +**And `clam_v3.rs` already answers which carving the real bake wants — it is +not the pair reading.** `RailSpec` carries TWO carvings, and the doc comment +records a measurement against them: the interleaved `X:Y` pair reading +(`v3_facet`, 6 levels, stride 2) *"fits only 44.25 % of paths"* on the +medcare bake, while the contiguous per-axis slab (`RailSpec::slab`, 12 levels, +stride 1) *"fits 99.62 % in twelve levels"*. That inverts the gate for G8: on +the **slab** carving a contiguous `u64` IS one axis, the mixed-axis caveat +above evaporates, and a plain `lzcnt(bswap(·))` is exactly right. On the +**pair** carving it is a mixed-axis tiebreaker. So G8's consumer gate must +name the CARVING as well as the call site, and the measured 99.62 %/44.25 % +split says the slab is the likelier target. Do not build against the pair +reading on the strength of it being the zero-fallback default. + +**Stacking, not widening, is how the levels grow.** The same module: *"a class +that needs more than six levels does not widen a byte — it stacks a second +register, e.g. into the edge lane, and chains it (`RailSpec::stacked`). Depth +then runs 0..=12 over two registers, same hole rule, same arithmetic."* The +edge lane at `16..32` is explicitly contemplated as that continuation register +(*"`16..32` may be a continuation register (if the spec says so)"*). A G8 +column op over a stacked pair is therefore two operands chained, never one +wider one — which is also why no `u128` variant of G8 is proposed. Gate +before building: one named consumer call site that ranks by depth — the same +count rule G5 carries. + **The nightly arm is AHEAD of the stable arms, and it is the contract reference.** `src/simd_nightly/` carries **18 compare-to-mask pairs across every width**; the stable arms have a subset. So N2/N3 were not adding a