Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .claude/board/entries/2026-09-25-report-mask-set-coordinate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# 2026-09-25 — A SET coordinate for the report substrate: `CoordSpec::MaskSet`

**Status:** TEST-PINNED · OPEN (one pass per member, not one keyed pass)

## What landed
- `CoordSpec::MaskSet { base, count }`: member `m` is the resident mask `base + m`. It is the coordinate of a many-to-many axis — paperless-ngx's tags, the gap `tesseract-paperless::axes` named ("count per tag is one scalar fold per tag, not one pivot").
- A row lands in every member whose mask holds it (zero, one or several), so the dimension's cells do not sum to the selected population. The per-row test oracle (`tests/common`) now places a row in the cartesian product of its per-dimension memberships.
- Planned as `Provider::MaskPlanes`: each member is one mask plane read in place, always a partition, never the fold key. A missing member mask is `UnknownMask`; `count == 0` is the new `EmptyMaskSet`.
- `CoordSpec::field()` now returns `Option<FieldId>` (a mask set reads no lane).

## Evidence
- `tests/mask_set.rs` (6): alone, crossed with an ordinal fold key (dense and sparse), under a selection — all against the oracle; per-tag counts equal the (row, tag) membership count and exceed the tagged rows; missing mask refused; empty set refused; explain names the set. The fixture asserts it contains untagged rows and rows with ≥ 2 tags.
- Disable runs, each red then restored: member filter reading the validity plane (3 tests fail); empty-set guard removed; missing mask read as the validity plane.

## Open
- Cost is one population pass per member (× the other partitions), exactly as before — the win is one plan, one result space and one render, not fewer passes. A keyed multi-membership fold (one pass for all members) would need a mask-RISC aggregation that does not exist.
3 changes: 2 additions & 1 deletion .claude/board/entries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
opposite directions; the stranding this convention prevents shows up in
exactly one of them, never both.

171 entries, 2026-08-06 .. 2026-09-25.
173 entries, 2026-08-06 .. 2026-09-25.

| date | entry id | finding | file |
|---|---|---|---|
Expand All @@ -34,6 +34,7 @@ exactly one of them, never both.
| 2026-09-25 | `tern3-six-view-window` | | [2026-09-25-tern3-six-view-window.md](2026-09-25-tern3-six-view-window.md) |
| 2026-09-25 | `tern2-two-level-lowering` | | [2026-09-25-tern2-two-level-lowering.md](2026-09-25-tern2-two-level-lowering.md) |
| 2026-09-25 | `strided-field-views-close-the-mask-risc-ir-gap` | | [2026-09-25-strided-field-views-close-the-mask-risc-ir-gap.md](2026-09-25-strided-field-views-close-the-mask-risc-ir-gap.md) |
| 2026-09-25 | `report-mask-set-coordinate` | | [2026-09-25-report-mask-set-coordinate.md](2026-09-25-report-mask-set-coordinate.md) |
| 2026-09-25 | `llvm-whole-stack-fold-ceiling` | | [2026-09-25-llvm-whole-stack-fold-ceiling.md](2026-09-25-llvm-whole-stack-fold-ceiling.md) |
| 2026-09-25 | `lance12-lancedb039-sweep` | | [2026-09-25-lance12-lancedb039-sweep.md](2026-09-25-lance12-lancedb039-sweep.md) |
| 2026-09-25 | `keep-fold` | | [2026-09-25-keep-fold.md](2026-09-25-keep-fold.md) |
Expand Down
101 changes: 98 additions & 3 deletions crates/lance-graph-report/src/batch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,13 @@
//! Names and labels are not stored here at all — they live in the catalog and
//! the CAM label store at the boundary.

use std::collections::HashMap;
use std::sync::Arc;

use lance_graph_mask_risc::{words_for, LaneRef};

use crate::ids::{FieldId, MaskId, SourceId};
use crate::ReportError;

/// One resident fixed-width lane.
#[derive(Debug, Clone)]
Expand Down Expand Up @@ -209,12 +211,34 @@ impl AbiBatch {
.map(|(i, c)| (i as u16, c))
}

/// Plane index of a resident mask.
/// Plane index of a resident mask, or `None` when no resident mask has
/// `id`. A mask past plane `u16::MAX` has no plane index and also reads
/// as `None`; [`Self::resolve_plane`] tells the two apart.
pub fn plane_of(&self, id: MaskId) -> Option<u16> {
self.masks
self.resolve_plane(id).ok()
}

/// Plane index of a resident mask. Refuses an unknown id, and refuses a
/// mask whose position does not fit a plane index rather than wrapping
/// onto another plane.
pub(crate) fn resolve_plane(&self, id: MaskId) -> Result<u16, ReportError> {
let i = self
.masks
.iter()
.position(|(m, _)| *m == id)
.map(|i| i as u16)
.ok_or(ReportError::UnknownMask(id))?;
u16::try_from(i).map_err(|_| ReportError::TooManyPlanes)
}

/// Every resident mask's position, built in one pass, for resolving many
/// ids at once without a scan per id. Positions are unchecked; convert
/// with `u16::try_from` at the point of use.
pub(crate) fn mask_positions(&self) -> HashMap<MaskId, usize> {
self.masks
.iter()
.enumerate()
.map(|(i, (m, _))| (*m, i))
.collect()
}

/// Borrowed plane views for the evaluator: O(columns + masks) pointers.
Expand All @@ -239,3 +263,74 @@ impl AbiBatch {
}
}
}

#[cfg(test)]
mod tests {
use super::*;
use crate::{AxisRole, CoordSpec, Measure, PlannerPolicy, ReportPlan, SourceRef};

/// A one-row batch holding validity plus `n` masks `M1..=Mn`. The masks
/// are pushed directly: `with_mask`'s duplicate check is a scan per call,
/// which would make building 65,536 of them quadratic.
fn wide(n: u32) -> AbiBatch {
let mut b = AbiBatch::new(SourceId(1), 1, 1);
let words: Arc<[u64]> = vec![0u64].into();
b.masks
.extend((1..=n).map(|i| (MaskId(i), Arc::clone(&words))));
b
}

#[test]
fn a_mask_past_plane_u16_max_is_refused_not_wrapped() {
let b = wide(65_536);
// Position 65_535 is the last addressable plane.
assert_eq!(b.resolve_plane(MaskId(65_535)), Ok(65_535));
// Position 65_536 would wrap to plane 0, the validity plane.
assert_eq!(
b.resolve_plane(MaskId(65_536)),
Err(ReportError::TooManyPlanes)
);
assert_eq!(b.plane_of(MaskId(65_536)), None);
assert_eq!(
b.resolve_plane(MaskId(70_000)),
Err(ReportError::UnknownMask(MaskId(70_000)))
);
}

/// The set coordinate resolves every member, so its last member is the
/// one that would have landed on the validity plane.
#[test]
fn a_mask_set_reaching_past_plane_u16_max_is_refused() {
let b = wide(65_536);
let plan = ReportPlan::over(SourceRef {
id: SourceId(1),
generation: 1,
})
.axis(
CoordSpec::MaskSet {
base: MaskId(1),
count: 65_536,
},
AxisRole::Row,
)
.measure(Measure::count());
assert_eq!(
plan.explain(&b, &PlannerPolicy::default()).unwrap_err(),
ReportError::TooManyPlanes
);
// One member fewer stays inside the addressable planes.
let ok = ReportPlan::over(SourceRef {
id: SourceId(1),
generation: 1,
})
.axis(
CoordSpec::MaskSet {
base: MaskId(1),
count: 65_535,
},
AxisRole::Row,
)
.measure(Measure::count());
assert!(ok.explain(&b, &PlannerPolicy::default()).is_ok());
}
}
58 changes: 47 additions & 11 deletions crates/lance-graph-report/src/exec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,12 @@ pub enum Provider {
/// Lane index of the bucketed field.
lane: u16,
},
/// A set coordinate: member `m` is the resident mask plane `planes[m]`,
/// read in place. Never the fold key — a row may sit in several members.
MaskPlanes {
/// Plane index of each member's mask.
planes: Vec<u16>,
},
}

/// One canonical dimension as planned.
Expand Down Expand Up @@ -232,8 +238,19 @@ fn member_filter(d: &DimPlan, m: u32) -> Filter {
_ => Filter::And(parts),
}
}
(Provider::DerivedBucket { .. }, CoordSpec::Field(_)) => {
unreachable!("bucket provider on a field")
(Provider::MaskPlanes { planes }, _) => Filter::Plane(Mask(planes[m as usize])),
(Provider::DerivedBucket { .. }, _) => {
unreachable!("bucket provider on a non-bucket coordinate")
}
}
}

/// The lane of the fold key. Only an ordinal lane is ever chosen as the key.
fn fold_key_lane(d: &DimPlan) -> u16 {
match d.provider {
Provider::OrdinalLane { lane } => lane,
Provider::DerivedBucket { .. } | Provider::MaskPlanes { .. } => {
unreachable!("the fold key is always an ordinal lane")
}
}
}
Expand All @@ -245,7 +262,28 @@ fn resolve(
) -> Result<Resolved, ReportError> {
let mut dims = Vec::with_capacity(key.coords.len());
for c in &key.coords {
let f = c.field();
if let CoordSpec::MaskSet { base, count } = c {
if *count == 0 {
return Err(ReportError::EmptyMaskSet(*base));
}
// One lookup table for the whole set: a scan per member would make
// resolution quadratic in the set size.
let positions = batch.mask_positions();
let planes = (0..*count)
.map(|m| {
let id = c.member_mask(m).ok_or(ReportError::TooManyPlanes)?;
let i = *positions.get(&id).ok_or(ReportError::UnknownMask(id))?;
u16::try_from(i).map_err(|_| ReportError::TooManyPlanes)
})
Comment thread
AdaWorldAPI marked this conversation as resolved.
.collect::<Result<Vec<_>, _>>()?;
dims.push(DimPlan {
coord: c.clone(),
domain: *count,
provider: Provider::MaskPlanes { planes },
});
continue;
}
let f = c.field().expect("a lane coordinate names a field");
let (lane, col) = batch.column(f).ok_or(ReportError::UnknownField(f))?;
let dim = match c {
CoordSpec::Field(_) => DimPlan {
Expand All @@ -263,6 +301,7 @@ fn resolve(
provider: Provider::DerivedBucket { lane },
}
}
CoordSpec::MaskSet { .. } => unreachable!("handled above"),
};
dims.push(dim);
}
Expand Down Expand Up @@ -462,10 +501,7 @@ impl ReportPlan {
} else {
base
};
let key_lane = r.fold_key.map(|i| match r.dims[i].provider {
Provider::OrdinalLane { lane } => lane,
Provider::DerivedBucket { lane } => lane,
});
let key_lane = r.fold_key.map(|i| fold_key_lane(&r.dims[i]));
let first_pass = r
.states
.iter()
Expand Down Expand Up @@ -555,9 +591,7 @@ impl ReportPlan {
lanes: &lanes,
};

let key_lane = r.fold_key.map(|i| match r.dims[i].provider {
Provider::OrdinalLane { lane } | Provider::DerivedBucket { lane } => lane,
});
let key_lane = r.fold_key.map(|i| fold_key_lane(&r.dims[i]));
let key_domain = r.fold_key.map_or(1, |i| r.dims[i].domain as usize);

// One program run: lowers, sizes branch-private scratch, executes.
Expand Down Expand Up @@ -717,7 +751,9 @@ impl ReportPlan {
stats.tile_mask_ops += prog.ops.len() as u64;
radices.push((0..d.domain).filter(|&m| buf[m as usize] > 0).collect());
}
Provider::DerivedBucket { .. } => radices.push((0..d.domain).collect()),
Provider::DerivedBucket { .. } | Provider::MaskPlanes { .. } => {
radices.push((0..d.domain).collect());
}
}
}
let passes: u128 = radices.iter().map(|r| r.len() as u128).product();
Expand Down
17 changes: 16 additions & 1 deletion crates/lance-graph-report/src/explain.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@ impl fmt::Display for PhysicalPlan {
format!("derived bucket (origin {origin}, width {width}; not materialized)")
}
(Provider::DerivedBucket { .. }, _) => "derived".to_string(),
(Provider::MaskPlanes { planes }, _) => {
format!(
"mask set ({} resident masks; a row may sit in several)",
planes.len()
)
}
};
let role = if self.fold_key == Some(i) {
"fold key"
Expand All @@ -57,7 +63,7 @@ impl fmt::Display for PhysicalPlan {
writeln!(
f,
" {} {prov} · domain {} · {role}",
d.coord.field(),
coord_name(&d.coord),
d.domain
)?;
}
Expand Down Expand Up @@ -85,3 +91,12 @@ impl fmt::Display for PhysicalPlan {
write!(f, "Materialization: terminal only")
}
}

fn coord_name(c: &CoordSpec) -> String {
match c {
CoordSpec::MaskSet { base, count } => {
format!("{base}..M{}", u64::from(base.0) + u64::from(*count))
}
_ => c.field().map_or_else(String::new, |f| f.to_string()),
}
}
2 changes: 2 additions & 0 deletions crates/lance-graph-report/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ pub enum ReportError {
NotACoordinate(FieldId),
/// A bucket coordinate over a non-`I32` field or with width <= 0.
BadBucket(FieldId),
/// A mask-set coordinate with no members.
EmptyMaskSet(MaskId),
/// A measure over a non-`I32` field.
NotAMeasure(FieldId),
/// The plan was minted against another source or generation.
Expand Down
34 changes: 29 additions & 5 deletions crates/lance-graph-report/src/plan.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
//! computed for one is reinterpreted for the other without a rescan
//! (falsifiers F13 / F14).

use crate::ids::{FieldId, SourceId};
use crate::ids::{FieldId, MaskId, SourceId};
use crate::selection::Selection;

/// The role an axis plays in the presented coordinate system.
Expand All @@ -29,7 +29,7 @@ pub enum AxisRole {

/// A coordinate provider — WHERE in the aggregate space a row lands.
///
/// Two providers, and neither knows what it measures:
/// Three providers, and none knows what it measures:
///
/// * [`CoordSpec::Field`] — a resident `U32` code lane with a declared domain.
/// It can serve as the fold KEY (the one dimension a single substrate pass
Expand All @@ -39,6 +39,12 @@ pub enum AxisRole {
/// a population lane: each member is a pair of tile-evaluated range
/// predicates (`origin + b·width <= v < origin + (b+1)·width`) applied
/// during the fold. A value outside every bucket lies outside the domain.
/// * [`CoordSpec::MaskSet`] — a SET coordinate: member `m` is the resident
/// mask `base + m`. It is the coordinate of a many-to-many axis (a document
/// carries several tags). A row lands in EVERY member whose mask holds it,
/// so the cells of this dimension do not sum to the selected population —
/// a row in two tags counts once in each, and a row in none lands nowhere.
/// Each member is one mask plane read in place; it is never the fold key.
#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub enum CoordSpec {
/// A resident ordinal field.
Expand All @@ -54,13 +60,31 @@ pub enum CoordSpec {
/// Number of buckets.
count: u32,
},
/// A set coordinate over `count` resident masks `base .. base + count`.
MaskSet {
/// The mask of member 0.
base: MaskId,
/// Number of members.
count: u32,
},
}

impl CoordSpec {
/// The field this coordinate reads.
pub fn field(&self) -> FieldId {
/// The field this coordinate reads, or `None` for a [`CoordSpec::MaskSet`],
/// which reads masks rather than a lane.
pub fn field(&self) -> Option<FieldId> {
match self {
CoordSpec::Field(f) | CoordSpec::Bucket { field: f, .. } => Some(*f),
CoordSpec::MaskSet { .. } => None,
}
}

/// The mask of member `m` of a [`CoordSpec::MaskSet`], or `None` for a
/// lane coordinate or a member past the set.
pub fn member_mask(&self, m: u32) -> Option<MaskId> {
match self {
CoordSpec::Field(f) | CoordSpec::Bucket { field: f, .. } => *f,
CoordSpec::MaskSet { base, count } if m < *count => base.0.checked_add(m).map(MaskId),
_ => None,
}
}
}
Expand Down
3 changes: 3 additions & 0 deletions crates/lance-graph-report/src/render.rs
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,9 @@ impl Terminal<'_> {
let lo = origin + width * i64::from(m);
format!("[{lo},{})", lo + width)
}
CoordSpec::MaskSet { .. } => c
.member_mask(m)
.map_or_else(|| m.to_string(), |id| id.to_string()),
}
}

Expand Down
4 changes: 1 addition & 3 deletions crates/lance-graph-report/src/selection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -178,9 +178,7 @@ impl Selection {
}
Filter::Cmp(Col(0), Cmp::Range { lo: r.lo, hi: r.hi })
}
Selection::Mask(id) => Filter::Plane(Mask(
batch.plane_of(*id).ok_or(ReportError::UnknownMask(*id))?,
)),
Selection::Mask(id) => Filter::Plane(Mask(batch.resolve_plane(*id)?)),
Selection::Predicate(p) => lower_predicate(p, batch)?,
Selection::And(a, b) => Filter::and([a.lower(batch)?, b.lower(batch)?]),
Selection::Or(a, b) => Filter::or([a.lower(batch)?, b.lower(batch)?]),
Expand Down
Loading
Loading