dir-sim: bounded numeric substrate — 64k u16 ordinals, Dn128, ValueId

.claude/board/entries/2026-10-05-text-to-numeric-boundary-inventory.md

@@ -0,0 +1,31 @@
+# 2026-10-05 — text → numeric id → Quack → mask-risc: one pattern, four id semantics
+
+## MEASURED — who runs the pattern (production callers traced)
+
+| consumer | text in | binder | numeric id | Quack in prod? | final primitive | text during execution |
+|---|---|---|---|---|---|---|
+| lance-graph-report | field name, categorical label | `Catalog` → `FieldId`; `CamLabels::ordinal` (CAM over `ContentId`) | `FieldId`, per-field CAM ordinal | yes (`exec.rs` calls `lance_graph_quack::lower`) | `Pred::EqU32` via `Cmp::EqU32` | none (`string_fence.rs`; counters in `reference_workload.rs`) |
+| lance-graph-sap | CATS columns | `bind.rs` first-occurrence dict per batch (forward map discarded) | batch-local `u32` code, 0 = NULL; employee via `numc` (parsed, not dict) | yes (`CatsQuery::prepare`) | `EqU32` / `GeI32` / `LeI32` + `GroupSumI32` | none; reverse labels kept for egress only |
+| lance-graph-java (lgj-abi) | none (static Java field constants) | compile-time `LaneId` | lane index + i64 operand | **no** — `plan_lower.rs` (quack is a dev-dependency) | same `Pred` vocabulary (`EqU32`, `GtI32`, `MatchU32`, …) | none |
+| lance-graph-dir-sim | UPN / SMTP, query literal | `Dicts::intern` / `key_lookup` | store-lifetime `ValueId` / `KeyId` | yes | `Pred::EqU32`, `MatchFacet16Strided`, `Gather` | none (`string_fence.rs`, `where_eq.rs`) |
+
+Java's `lowering_convergence` compares ANSWERS (row counts / group sums) over 28 combine vectors and 9 opcodes, not Programs. Moving Java onto Quack would be convergence, not invention.
+
+## FINDING — ids are not interchangeable
+
+- **`ContentId`** (contract): content identity, fnv1a64 of exact bytes. Process-independent, survives reopen, legal anywhere. It is u64, and it has no normalization.
+- **report CAM ordinal**: a per-field category. `rename` re-points the label and keeps the ordinal. That is presentation identity, safe because a coordinate means the category, not the text.
+- **report `FieldId`**: schema/catalog identity.
+- **SAP code**: one bound batch only (`CatsQuery` borrows the batch). It cannot be persisted, and a text literal cannot be resolved to it after binding.
+- **dir-sim `ValueId`**: the exact text, shared across attributes, append-only for the store's lifetime. A different text is a different value, and changing it is a `SetAttribute`. It is persisted in `Change` / `ExecutionPlan` / `Precondition`.
+- **dir-sim `KeyId`**: the normalized comparison form, with the same lifetime. It is used only for comparison: uniqueness and rename ordering.
+- **OGAR `ValuePool` `StrRef`**: a per-batch byte offset, not deduplicated. It is not an identity.
+- **Quack `Col`**: a lane position inside one `Planes`.
+
+Consequence: replacing `ValueId` with the CAM ordinal would be wrong. A CAM rename keeps the id and changes the text, so a plan's compare-and-set would silently target a different string.
+
+## OPEN
+
+- Should `ValueId` converge onto `ContentId` (content identity, reopen-stable)? That would need u64 lanes and a collision policy. Decision pending.
+- SAP has no query-time literal → code path, because its forward map is dropped at bind.
+- Java production still lowers outside Quack.

.claude/board/entries/README.md

@@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
 opposite directions; the stranding this convention prevents shows up in
 exactly one of them, never both.
 
-213 entries, 2026-08-06 .. 2026-10-04.
+214 entries, 2026-08-06 .. 2026-10-05.
 
 | date | entry id | finding | file |
 |---|---|---|---|
+| 2026-10-05 | `text-to-numeric-boundary-inventory` |  | [2026-10-05-text-to-numeric-boundary-inventory.md](2026-10-05-text-to-numeric-boundary-inventory.md) |
 | 2026-10-04 | `D-LXC-22` |  | [2026-10-04-wordnet-clam-chaoda-scope-and-grammar-read-params.md](2026-10-04-wordnet-clam-chaoda-scope-and-grammar-read-params.md) |
 | 2026-10-04 | `D-HPS-1` |  | [2026-10-04-spog-slab-hotplug-resolution.md](2026-10-04-spog-slab-hotplug-resolution.md) |
 | 2026-10-04 | `D-HPS-2` |  | [2026-10-04-resolve-once-population-execution.md](2026-10-04-resolve-once-population-execution.md) |

crates/lance-graph-dir-sim/src/exec.rs

@@ -58,6 +58,12 @@ impl Kept {
         materialize_rows(&self.bits, self.n_rows)
     }
 
+    /// Add one row (a delta-sized correction, e.g. an overridden value).
+    pub(crate) fn set(&mut self, row: usize) {
+        debug_assert!(row < self.n_rows);
+        self.bits[row / 64] |= 1 << (row % 64);
+    }
+
     /// `self`'s rows followed by `tail`'s rows offset by `self`'s width —
     /// one result over a base relation and its delta rows.
     pub(crate) fn concat(mut self, tail: &Kept) -> Kept {

crates/lance-graph-dir-sim/src/lib.rs

@@ -7,14 +7,17 @@
 //! OGAR owns the meaning (`ogar-dir-sim`: `Change`, provenance, `Violation`,
 //! `ExecutionPlan`); this crate owns execution:
 //!
-//! * [`snapshot`] — one observation as SoA lanes and bit planes, `Guid128`
-//!   sorted so the ordinal is the index; strings in store dictionaries.
+//! * [`snapshot`] — one observation as two populations (users, groups) of
+//!   SoA lanes and bit planes, each at most 65,536 nodes with its own `u16`
+//!   ordinal space; sparse `user × group` membership; [`Dn128`] hierarchy;
+//!   strings only in the store's cold label/value table, as [`ValueId`]s.
 //! * [`view`] — a version = shared `Arc<Snapshot>` + delta-sized overlay.
 //! * [`rule`] — pure population rules over a borrowed [`View`].
 //! * [`validate`] — invariants as Quack programs (`Semijoin` anti-joins,
 //!   `GroupReduce` counts).
 //! * [`store`] — append-only versions, tags, diff, plan, audit.
-//! * [`observe`] — `ogar-ad` records → observation.
+//! * [`observe`] — `ogar-ad` records → observation (`OuHhtl` → `Dn128`,
+//!   failing closed).
 //!
 //! No network, process or file I/O. Nothing writes to AD, Entra, Exchange,
 //! LDAP or PowerShell. `#![forbid(unsafe_code)]`.
@@ -30,81 +33,151 @@ pub mod validate;
 pub mod view;
 
 pub use exec::Kept;
+pub use ogar_dir_sim::{KeyId, ValueId};
 pub use rule::{member_counts, GrantGroup, ImplyGroup, Rule, SetPrimarySmtp};
 pub use snapshot::{
-    pack_ou, BuildError, Dict, Dicts, NodeKind, Observation, ObservedNode, Snapshot, NONE,
+    BuildError, Dict, DictCounters, Dicts, GroupOrdinal, NodeKind, Observation, ObservedNode,
+    Population, Snapshot, UserOrdinal, MAX_GROUPS, MAX_USERS, NONE,
 };
 pub use store::{Rejection, SimError, VersionStore};
 pub use view::{ApplyError, View};
 
-use lance_graph_mask_risc::{Foreign, LaneRef, Planes};
+use lance_graph_mask_risc::{words_for, Foreign, LaneRef, Planes, Program, StridedRef};
 use lance_graph_quack::{Cmp, Col, Filter, Mask};
-use ogar_dir_core::OuHhtl;
+use ogar_dir_core::{DirectoryScope, Dn128};
+use ogar_dir_sim::Attribute;
 
-/// A subtree prefix deeper than the packed lane (4 levels) can address.
+/// The program behind [`users_with_key`]: plane 0 = candidate users,
+/// lane 0 = an attribute's key lane, one `EqU32` on the key. It holds only
+/// numbers: the literal was resolved before it was built.
+pub fn key_eq_program(key: KeyId) -> Program {
+    exec::program(
+        Filter::and([
+            Filter::plane(Mask(0)),
+            Filter::cmp(Col(0), Cmp::EqU32(key.0)),
+        ]),
+        lance_graph_quack::Agg::Rows,
+    )
+}
+
+/// `WHERE <attribute> = <literal>` over the existing users of a version, as
+/// a bitmap over user ordinals. The literal arrives as a [`KeyId`] —
+/// resolved once at the boundary ([`Dicts::key_lookup`]) — so execution is
+/// one [`key_eq_program`] over the base key lane (overridden and deleted
+/// users gated out), the same program over the created users' key lane,
+/// and a delta-sized check of the overrides. No string is read.
+pub fn users_with_key(v: &View<'_>, a: Attribute, key: KeyId) -> Kept {
+    let p = key_eq_program(key);
+    let s = v.snap;
+    let (pop, ov) = v.pop(NodeKind::User);
+    let base_key = match a {
+        Attribute::Upn => &pop.upn_key,
+        Attribute::PrimarySmtp => &pop.smtp_key,
+    };
+    let mut live = v.base_live(NodeKind::User, &pop.all).into_owned();
+    for o in ov.overrides(a).keys() {
+        snapshot::clear_bit(&mut live, usize::from(*o));
+    }
+    let run = |plane: &[u64], lane: &[u32]| {
+        let lanes = [LaneRef::U32(lane)];
+        let masks: [&[u64]; 1] = [plane];
+        exec::keep(
+            &p,
+            &Planes {
+                n_rows: lane.len(),
+                masks: &masks,
+                lanes: &lanes,
+            },
+            &Foreign::NONE,
+        )
+    };
+    let mut base = run(&live, base_key);
+    for (o, (_, k)) in ov.overrides(a) {
+        if *k == key.0 {
+            base.set(usize::from(*o));
+        }
+    }
+    debug_assert_eq!(base_key.len(), s.users.len());
+    let created = ov.created.key(a);
+    base.concat(&run(&snapshot::ones(created.len()), created))
+}
+
+/// A subtree query in a directory the version does not describe. Codes are
+/// only meaningful under their scope, so the query is refused, not run.
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
-pub struct SubtreeTooDeep(pub usize);
+pub struct ScopeMismatch {
+    /// The version's scope.
+    pub version: DirectoryScope,
+    /// The query's scope.
+    pub query: DirectoryScope,
+}
 
-/// Nodes located in the OU subtree `prefix` (ancestor-or-self), as a node
-/// bitmap over the version's ordinals — one ternary match on the packed OU
-/// lane (`Cmp::MatchU64`), no DN strings. The same program runs over the
-/// base lane (deleted nodes gated out) and over the created nodes' lane
-/// (delta-sized); the two kept sets are concatenated, never fed onward.
-/// Prefixes up to depth 4 are exact; deeper ones are refused.
-pub fn subtree(v: &View<'_>, prefix: &OuHhtl) -> Result<exec::Kept, SubtreeTooDeep> {
-    let d = prefix.depth();
-    if d > 4 {
-        return Err(SubtreeTooDeep(d));
+/// Nodes of `kind` located in the subtree of `prefix` (ancestor-or-self),
+/// as a bitmap over that population's ordinals.
+///
+/// One program, no strings: `located ∧ depth ≥ d ∧ dn ⊇ prefix`, where the
+/// last term is a 16-byte ternary match (`Cmp::MatchFacet16Strided`) read
+/// in place over the population's `[[u8; 16]]` lane. The depth gate is what
+/// keeps a shallower node whose zero tail happens to equal the prefix out.
+/// The same program runs over the base lane (deleted nodes gated out) and
+/// over the created nodes' lane (delta-sized); the two kept sets are
+/// concatenated, never fed onward.
+pub fn subtree(
+    v: &View<'_>,
+    scope: DirectoryScope,
+    kind: NodeKind,
+    prefix: &Dn128,
+) -> Result<Kept, ScopeMismatch> {
+    if scope != v.snap.scope {
+        return Err(ScopeMismatch {
+            version: v.snap.scope,
+            query: scope,
+        });
     }
-    let care = if d == 0 { 0 } else { u64::MAX << (64 - 16 * d) };
+    let (pattern, care) = prefix.subtree_mask();
     let p = exec::program(
         Filter::and([
             Filter::plane(Mask(0)),
-            Filter::cmp(
-                Col(0),
-                Cmp::MatchU64 {
-                    pattern: pack_ou(prefix),
-                    care,
-                },
-            ),
+            Filter::cmp(Col(0), Cmp::GeI32(prefix.depth() as i32)),
+            Filter::cmp(Col(1), Cmp::MatchFacet16Strided { pattern, care }),
         ]),
         lance_graph_quack::Agg::Rows,
     );
-    let s = v.snap;
-    let present = v.base_live(&s.ou_present);
-    let lanes = [LaneRef::U64(&s.ou_hi)];
-    let masks: [&[u64]; 1] = [&present];
-    let base = exec::keep(
-        &p,
-        &Planes {
-            n_rows: s.len(),
-            masks: &masks,
-            lanes: &lanes,
-        },
-        &Foreign::NONE,
-    );
-    let cr = &v.ov.created;
-    let packed: Vec<u64> = cr
-        .ou
-        .iter()
-        .map(|o| o.as_ref().map_or(0, pack_ou))
-        .collect();
-    let mut located = vec![0u64; lance_graph_mask_risc::words_for(cr.ou.len())];
-    for (i, o) in cr.ou.iter().enumerate() {
-        if o.is_some() {
+    let run = |present: &[u64], depth: &[i32], dn: &[[u8; 16]]| {
+        let lanes = [
+            LaneRef::I32(depth),
+            LaneRef::Strided(StridedRef {
+                bytes: dn.as_flattened(),
+                first_offset: 0,
+                stride: 16,
+                records: dn.len(),
+            }),
+        ];
+        let masks: [&[u64]; 1] = [present];
+        exec::keep(
+            &p,
+            &Planes {
+                n_rows: dn.len(),
+                masks: &masks,
+                lanes: &lanes,
+            },
+            &Foreign::NONE,
+        )
+    };
+    let (pop, ov) = v.pop(kind);
+    let present = v.base_live(kind, &pop.dn_present);
+    let base = run(&present, &pop.dn_depth, &pop.dn);
+
+    let cr = &ov.created.dn;
+    let mut located = vec![0u64; words_for(cr.len())];
+    let (mut depth, mut dn) = (Vec::with_capacity(cr.len()), Vec::with_capacity(cr.len()));
+    for (i, d) in cr.iter().enumerate() {
+        if d.is_some() {
             located[i / 64] |= 1 << (i % 64);
         }
+        let d = d.unwrap_or(Dn128::ROOT);
+        depth.push(d.depth() as i32);
+        dn.push(d.bytes());
     }
-    let lanes = [LaneRef::U64(&packed)];
-    let masks: [&[u64]; 1] = [&located];
-    let created = exec::keep(
-        &p,
-        &Planes {
-            n_rows: cr.ou.len(),
-            masks: &masks,
-            lanes: &lanes,
-        },
-        &Foreign::NONE,
-    );
-    Ok(base.concat(&created))
+    Ok(base.concat(&run(&located, &depth, &dn)))
 }

crates/lance-graph-dir-sim/src/observe.rs

@@ -3,13 +3,22 @@
 //! The ingestion boundary: values are read out of the record's value pool
 //! once and handed to [`Snapshot::build`](crate::Snapshot::build) for
 //! interning. "Active" is derived from `userAccountControl` bit `0x2`
-//! (ACCOUNTDISABLE); the primary SMTP is the `SMTP:` proxy; the OU-HHTL is
-//! taken as-is (never a DN string). Memberships are relations that `ogar-ad`
-//! records do not carry; the caller adds observed ones.
+//! (ACCOUNTDISABLE); the primary SMTP is the `SMTP:` proxy; the location is
+//! the record's `OuHhtl` (the ingress wire format) converted to a [`Dn128`],
+//! never a DN string. A parent with more than 256 children cannot be a
+//! `Dn128` and the whole observation is refused — never hashed or truncated.
+//! Memberships are relations that `ogar-ad` records do not carry; the
+//! caller adds observed ones.
+//!
+//! **Open, recorded, not decided here:** a user record with no
+//! `userAccountControl` value is read as **enabled** (`is_none_or`). Whether
+//! an absent flag should mean enabled, disabled or "unknown" — and how an AD
+//! and an Entra observation of the same person are merged — is an open
+//! policy question; this module does not choose for it.
 
 use crate::snapshot::{NodeKind, Observation, ObservedNode};
 use ogar_ad::{AdKind, SCHEMA_V1};
-use ogar_dir_core::{DirRecord, ValuePool};
+use ogar_dir_core::{DirRecord, DirectoryScope, Dn128, Dn128Error, Guid128, ValuePool};
 
 const UAC_ACCOUNTDISABLE: u32 = 0x2;
 
@@ -21,15 +30,45 @@ fn slot(name: &str) -> usize {
         .expect("ogar-ad schema v1")
 }
 
-/// Users and groups among `records` (other kinds are skipped).
-pub fn from_ad(records: &[DirRecord], pool: &ValuePool) -> Observation {
+/// Why a record could not be observed. Either way the whole observation is
+/// refused: nothing is silently dropped or merged.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum ObserveError {
+    /// The record's `OuHhtl` is not a `Dn128` (e.g. a 257th child).
+    Location {
+        /// The record.
+        node: Guid128,
+        /// Why.
+        error: Dn128Error,
+    },
+    /// The record belongs to another directory than the observation. A
+    /// `Dn128` carries no scope (it is external context), so a record from
+    /// another domain or tenant would be indistinguishable from a local one.
+    ForeignScope {
+        /// The record.
+        node: Guid128,
+        /// The record's own scope.
+        scope: DirectoryScope,
+    },
+}
+
+/// Users and groups among `records` (other kinds are skipped), located in
+/// `scope`. Every user or group record must carry that scope.
+pub fn from_ad(
+    scope: DirectoryScope,
+    records: &[DirRecord],
+    pool: &ValuePool,
+) -> Result<Observation, ObserveError> {
     let text = |r: &DirRecord, name: &str| {
         r.str_ref(slot(name))
             .and_then(|s| pool.get(s))
             .and_then(|b| std::str::from_utf8(b).ok())
             .map(str::to_string)
     };
-    let mut obs = Observation::default();
+    let mut obs = Observation {
+        scope,
+        ..Observation::default()
+    };
     for r in records {
         let kind = match r.object_kind() {
             k if k == AdKind::User as u16 => NodeKind::User,
@@ -44,16 +83,28 @@ pub fn from_ad(records: &[DirRecord], pool: &ValuePool) -> Observation {
                     .filter_map(|v| std::str::from_utf8(v).ok())
                     .find_map(|v| v.strip_prefix("SMTP:").map(str::to_string))
             });
+        let node = r.node_guid();
+        if r.scope_guid() != scope.0 {
+            return Err(ObserveError::ForeignScope {
+                node,
+                scope: DirectoryScope(r.scope_guid()),
+            });
+        }
+        let dn = r
+            .ou_hhtl()
+            .map(|h| Dn128::from_ou_hhtl(&h))
+            .transpose()
+            .map_err(|error| ObserveError::Location { node, error })?;
         obs.nodes.push((
-            r.node_guid(),
+            node,
             ObservedNode {
                 kind,
                 active: r.num(0).is_none_or(|uac| uac & UAC_ACCOUNTDISABLE == 0),
                 upn: text(r, "userPrincipalName"),
                 primary_smtp,
-                ou: r.ou_hhtl(),
+                dn,
             },
         ));
     }
-    obs
+    Ok(obs)
 }