Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .claude/board/entries/2026-09-29-grouped-cross-moments-fold.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# 2026-09-29 — Grouped cross moments: Pearson / covariance / OLS / R² over masks

**Status:** MEASURED · DONE — ndarray `simd_masking_ops.rs`, `crates/lance-graph-mask-risc`, `crates/jc`

## What landed
- ndarray: `GroupCrossPowerSums { n, sum_x, sum_y: i64, sum_x2, sum_y2, sum_xy: i128 }` + `masked_group_cross_moments_i32{,_via,_pair}` over the same generic `group_walk`. `x_moments()`/`y_moments()` return the exact univariate `GroupPowerSums`; a test pins them equal to the univariate fold. Bounds: squares ∈ [0, 2^62], products ∈ [−2^62+2^31, 2^62]; the `i64` sums bind at 2^32 rows/group; the `i128` sums cannot overflow at any `u64` row count.
- mask-risc: `Terminal::GroupCrossPowerSumsI32 { mask, key, x, y }` → `Out::CrossPowerSums`.
- jc: `pearson_from_cross_power_sums`, `sample_covariance_from_cross_power_sums` (n−1), `simple_regression_from_cross_power_sums` (`SimpleRegression { slope, intercept }`), `r_squared_from_cross_power_sums` (`multiple_r_squared`'s k=1 contract). Shared tails extracted: `pearson_from_centered`, `sample_cov_tail`, `r_squared_tail`. Centred sums formed exactly in `i128`, checked.

## Measured
48 layouts (n ≤ 100k): |Δr| ≤ 1.3e-14, |ΔR²| ≤ 4.1e-14, relative Δslope ≤ 2.8e-14 vs the materialized path. Huge offset + tiny spread (x ≈ ±2·10⁹): fold path within 1e-15 of an independent exact reference. **Unlike ANOVA, the slice path does not fail here** — `pearson`/`multiple_r_squared` are two-pass. A naive f64 projection of the same moments returns NaN; that is what exact `i128` centring prevents (disable run: 2 tests fail).

## Redundant guards (disable runs, kept as explicit contract)
`pearson_from_cross_power_sums`'s `n < 2`, the regression's `cxx == 0`, and R²'s `cyy == 0` are each subsumed by the shared tail's zero/non-finite rejection. The first past-bound refusal fixture was vacuous (wrapped to negative centred squares); replaced with one that wraps to a plausible `r = 1`.

## Open
- Multi-membership in one pass: `lance-graph-report` lowers `CoordSpec::MaskSet` to one `Filter::Plane` pass per member tuple and per fold state. The single-group limit is in `GroupKeyAddr::group_of` (one `Option<usize>` per row), in the IR's `GroupKey`, and in report lowering. Seam: a word-level walker `for each 64-row word: sel & plane_m → fold hits into out[m]` over existing planes — same plane traffic as K passes, value-lane traffic of the union instead of the sum.

**Renamed 2026-09-30 (rebase onto ndarray master):** upstream shipped `PowerSums { n, sum, sum_sq: u128 }` — the same record as `GroupPowerSums`. The duplicate was dropped: consumers use `PowerSums` / `CrossPowerSums` and `masked_group_(cross_)power_sums_i32*`; `checked_merge` moved onto the upstream types. Square sums are now `u128`; jc converts with `i128::try_from`, and a value past `i128::MAX` is refused as past the bound.
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# 2026-09-29 — One-way ANOVA over a population mask from grouped moments

**Status:** MEASURED · DONE — ndarray `simd_masking_ops.rs`, `crates/lance-graph-mask-risc`, `crates/jc/src/stats.rs`

## What landed
- ndarray: `GroupPowerSums { n: u64, sum: i64, sum_sq: i128 }` and `masked_group_moments_i32{,_via,_pair}` — `(n, Σx, Σx²)` per group in one pass over the rows a mask selects, over the existing `group_walk` (now generic over its slot type). `checked_merge` is exact integer addition; exact up to `GROUP_MOMENTS_MAX_ROWS = 2^32` rows per group (the `i64` sum is the binding field).
- mask-risc: `Terminal::GroupPowerSumsI32 { mask, key: GroupKey, val }` → `Out::PowerSums`. Own terminal, not a `GroupFold` member (a `GroupFold` slot is one seeded `i64`). Refuses planes past `MASKED_SUM_I32_MAX_ROWS`. Partial extents: admitted since #1323's extent wiring — per-extent sinks merge by `checked_merge` (pinned in `tests/extent.rs`).
- jc: `anova_from_power_sums` / `eta_squared_from_power_sums`. F/p/η² policy is shared with `anova_one_way` / `eta_squared` via `anova_from_ss` / `eta_from_ss`. Sums of squares come from exact `i128` quantities; no two large floats are subtracted.

## Measured
Mask → fold → `anova_from_power_sums` vs materialized → `anova_one_way`, 42 non-degenerate layouts (n ≤ 100k, k ≤ 6, densities 20/128/256 of 256): max relative ΔF 4.2e-14, max |Δp| 4.4e-16, max |Δη²| 6.5e-16. Adversarial layout (group means ±2·10⁹, within spread 1): exact F = 3·10¹⁹; the moments path returns 3·10¹⁹, the slice path returns `None` (its `ss_t − ss_b` cancels the within-group SS to ≤ 0). A naive float `Q − S²/N` inside the moments path fails even the near-`i32::MAX` agreement test (ΔF ≈ 1e-3), so the exact forms are load-bearing.

## Open
- Multi-membership is NOT one pass: `GroupKey` resolves one group per row, and `lance-graph-report`'s `CoordSpec::MaskSet` executes one `Filter::Plane` program per member. Seam: a key address that walks the selected rows once and folds each row into every member plane that holds it.
- Bivariate `(n, Σx, Σy, Σx², Σy², Σxy)`: same walk, a second value lane in the closure, a wider slot type; not built.
- No early exit, and no `CausalEdge64` commit. ANOVA is non-monotone under future rows.

**Renamed 2026-09-30 (rebase onto ndarray master):** upstream shipped `PowerSums { n, sum, sum_sq: u128 }` — the same record as `GroupPowerSums`. The duplicate was dropped: consumers use `PowerSums` / `CrossPowerSums` and `masked_group_(cross_)power_sums_i32*`; `checked_merge` moved onto the upstream types. Square sums are now `u128`; jc converts with `i128::try_from`, and a value past `i128::MAX` is refused as past the bound.
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# 2026-09-29 — perturbation-sim: Σθ = L⁺ Σp L⁺ through ndarray CovHighD::sandwich

**STATUS:** MEASURED · **Scope:** `crates/perturbation-sim/src/angle_cov.rs` (feature `pillar`), ndarray `hpc::pillar::cov_high_d`

## What landed
- ndarray `4d4ee17`: `CovHighD::from_symmetric_fn` + public `get`. Before this a consumer
could not build a CovHighD from data without re-spelling the packed index.
- lance-graph `bb6d044`: `angle_covariance::<N>(eig, Σp, rel_tol)`, the stochastic twin of
`pseudo_apply`. It is exact for DC flow in a fixed topology. It calls the Pillar-9 sandwich;
there is no local kernel.

## Measured (6-bus ring + chords)
- Against an f64 dense triple product: rel err 1.2e-7. This is the f32 floor.
- Rank-1 Σp = ppᵀ against `pseudo_apply` outer product: 2.3e-7. That path never forms L⁺.
- 40k-sample Monte Carlo of the deterministic solver: rel err 0.0009.
- Disable runs, each red under exactly its own test:
- M := I;
- symmetry guard removed;
- N-mismatch guard removed.

## Finding in ndarray
Every existing `sandwich` test used M = I. Under a deliberately broken kernel (M·Σ·Σ),
`sandwich_identity_is_identity` stayed green. The new dense non-identity test is the first
that can see an index/transpose defect in the Pillar-9 kernel.

## OPEN
- CovHighD is const-generic N; `Grid::n` is runtime. The caller picks N and it is checked.
A runtime-sized sandwich is an ndarray change, not taken.
- Line trips are a rank-1 update of L⁺ (LODF), not a sandwich. They are not wired.
- Line-flow covariance has a non-symmetric rectangular Jacobian, which sandwich cannot
express. Per-line variance is a 2-sparse quadratic form over Σθ, not built.
5 changes: 4 additions & 1 deletion .claude/board/entries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
opposite directions; the stranding this convention prevents shows up in
exactly one of them, never both.

208 entries, 2026-08-06 .. 2026-10-04.
211 entries, 2026-08-06 .. 2026-10-04.

| date | entry id | finding | file |
|---|---|---|---|
Expand Down Expand Up @@ -61,6 +61,9 @@ exactly one of them, never both.
| 2026-09-30 | `three-reference-sets-are-not-ordinal-aligned` | | [2026-09-30-three-reference-sets-are-not-ordinal-aligned.md](2026-09-30-three-reference-sets-are-not-ordinal-aligned.md) |
| 2026-09-30 | `deepnsm-v2-coverage-bands` | | [2026-09-30-deepnsm-v2-coverage-bands.md](2026-09-30-deepnsm-v2-coverage-bands.md) |
| 2026-09-30 | `cypher-mask-v2-is-a-replacement-not-a-phase` | | [2026-09-30-cypher-mask-v2-is-a-replacement-not-a-phase.md](2026-09-30-cypher-mask-v2-is-a-replacement-not-a-phase.md) |
| 2026-09-29 | `perturbation-sim-angle-covariance-via-cov-high-d` | | [2026-09-29-perturbation-sim-angle-covariance-via-cov-high-d.md](2026-09-29-perturbation-sim-angle-covariance-via-cov-high-d.md) |
| 2026-09-29 | `grouped-moments-fold-anova-over-masks` | | [2026-09-29-grouped-moments-fold-anova-over-masks.md](2026-09-29-grouped-moments-fold-anova-over-masks.md) |
| 2026-09-29 | `grouped-cross-moments-fold` | | [2026-09-29-grouped-cross-moments-fold.md](2026-09-29-grouped-cross-moments-fold.md) |
| 2026-09-29 | `deepnsm-v2-counted-pick-tag-deltas` | | [2026-09-29-deepnsm-v2-counted-pick-tag-deltas.md](2026-09-29-deepnsm-v2-counted-pick-tag-deltas.md) |
| 2026-09-26 | `deepnsm-v2-lexical-evidence-survives-routing` | | [2026-09-26-deepnsm-v2-lexical-evidence-survives-routing.md](2026-09-26-deepnsm-v2-lexical-evidence-survives-routing.md) |
| 2026-09-25 | `window-scheduling-and-two-level-ternlog` | | [2026-09-25-window-scheduling-and-two-level-ternlog.md](2026-09-25-window-scheduling-and-two-level-ternlog.md) |
Expand Down
8 changes: 8 additions & 0 deletions crates/jc/src/reliability.rs
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,14 @@ pub fn pearson(x: &[f64], y: &[f64]) -> Option<f64> {
sxx += dx * dx;
syy += dy * dy;
}
pearson_from_centered(sxy, sxx, syy)
}

/// Pearson's `r` from the centred co-moment and the two centred
/// second moments — the ONE place its degeneracy policy lives, shared by
/// [`pearson`] and `stats::pearson_from_cross_power_sums`. Any common positive
/// scale on all three (e.g. `n·S` instead of `S`) cancels.
pub(crate) fn pearson_from_centered(sxy: f64, sxx: f64, syy: f64) -> Option<f64> {
let denom = (sxx * syy).sqrt();
if denom == 0.0 || !denom.is_finite() {
// `denom == 0` → at least one series is constant. `denom == ∞` → the
Expand Down
Loading
Loading