Merge pull request #1017 from AdaWorldAPI/claude/bpe-tokenization-architecture-3xd4eh

the integration half of #1012: one receipt, three borrowed consumers

Author: AdaWorldAPI

.claude/board/AGENT_LOG.md

@@ -1,3 +1,54 @@
+## 2026-08-23 — the token-seam arc: four read-only research lanes, one probe, one vacuity audit
+
+- **Why:** an operator brief asked whether ONE versioned BPE tokenization can
+  simultaneously serve Tantivy lexical indexing, DeepNSM-v2 lexical/grammar
+  projection and an LSTM forward-prediction input surface, without
+  retokenizing, without rebuilding a DataFrame, and without a second cognitive
+  population — the integration half `E-TOKEN-BPE-CAN-FIT-NOT-YET-BUY-1` (#1012)
+  explicitly did not ask. Tiering: Opus on the main thread for the architecture,
+  the probe and every gate; Sonnet for the bounded read-only lanes; no worker
+  ran cargo (the orchestrator compiled centrally, per the shared-target rule).
+- **Lane A — DeepNSM-v2 lexical contract.** The decisive finding: the LIBRARY is
+  already tokenizer-free. `parse_to_spo(&[Tagged])` takes `(WordId, Pos)` and no
+  string; `split_whitespace`/`normalise` live only in the two examples. The seam
+  therefore needed no change to the crate. Also: `academic_20k.csv` is present
+  (20 845 rows, 18 559 distinct surface forms); `bible_vocab.txt` and the cam96
+  codebook/codes are ABSENT.
+- **Lane B — Polars / online-path falsifier.** Zero `polars` occurrences across
+  nine checkouts; every `DataFrame` mention is prose. `paperless-rs` and
+  `tesseract-rs` declare no arrow/datafusion/lance/lancedb. Also established
+  that `doc.v1` carries bbox/conf/leading_space and NO offset or span field —
+  which is the gap that blocks the seam on real scanned documents.
+- **Lane C — Tantivy indexing-path audit.** The indexer never reads
+  `offset_from`/`offset_to` outside its own tests; snippets re-tokenize STORED
+  text at query time; `PreTokenizedString` costs ≈ `4 + 2N` allocations because
+  `segment_writer` deep-clones the boxed value. That last number is why the seam
+  uses a custom tokenizer with one reused `Token` buffer.
+- **Lane D — SoA lane / continuation precedent.** No shipped token continuation
+  mechanism anywhere; the nearest in shape, `RailCarving::AxisSlab`, caps at 24
+  levels — below the measured p50 of 4 particles. `ValueTenant` has 16 variants
+  and none for text.
+- **The probe (main thread, Opus).** `PROBE-TOKEN-SEAM-1` — 37 gates, 13
+  disable-runs verified red-then-green, in `AdaWorldAPI/paperless-rs`
+  (`crates/paperless-token`), with `docs/TOKEN-SEAM-ARCHITECTURE.md` as the
+  bounded architecture. Result: `E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1`.
+- **Lane E — vacuity audit (Sonnet, read-only, adversarial).** Found FIVE holes
+  in the finished probe, all real: a gate checking byte counts but not span
+  counts (the CRLF bug that collapsed 300 spans into 1 would have re-passed), a
+  threshold true by construction, an assertion about a type signature rather
+  than behaviour, an unconditional prefix check, and an unexercised
+  ASCII-vs-Unicode whitespace divergence. All five fixed; four gained their own
+  disable-runs; the fifth is bounded by a measured count of zero.
+- **Two method failures worth the entry.** (1) Two disable-runs were themselves
+  wrong first — one relaxed a constant that does not bind on the fixture, one
+  targeted a mechanism the gate did not rest on — so both "passed" while proving
+  nothing. (2) An early disable batch reported "no failure" six times in a row
+  because the probe binary path was wrong and nothing ran at all. A null result
+  is a claim about the apparatus until proven otherwise.
+- **Outcome:** #1017 (board), plus a paperless-rs commit that is **committed
+  locally and BLOCKED from pushing** — the GitHub App has no access to
+  `AdaWorldAPI/paperless-rs` for this org, verified through both the session
+  proxy and a proxy-bypassed attempt.
 ## 2026-08-23 — autoattended R2IL wave: 4 Sonnet probe workers + 1 Sonnet scribe + 1 Opus synthesis + 1 Haiku guarded executor
 
 - **Why:** operator directive to run the pattern autonomously — "sonnet agents

.claude/board/EPIPHANIES.md

@@ -1,3 +1,154 @@
+## 2026-08-23 — E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1 — the integration half of #1012: one versioned tokenization drives Tantivy, DeepNSM-v2 and a forward surface with zero re-tokenization, and the four gaps that stand between that and a carrier
+
+**Status:** FINDING — [MEASURED] (`PROBE-TOKEN-SEAM-1`, 37 gates, **13
+disable-runs each verified red-then-green**; two committed real corpora — the
+in-tree KJV Genesis scene `PROBE-TOKEN-BPE-GEOMETRY-1` used, carried verbatim so
+the numbers are comparable, and Project Gutenberg's *Alice* split into 300
+paragraphs, 75 514 B). The probe lives in `AdaWorldAPI/paperless-rs`
+(`crates/paperless-token`, `docs/TOKEN-SEAM-ARCHITECTURE.md`) because it needs a
+Tantivy dependency this workspace does not carry; this entry records the
+**lance-graph-side** findings.
+**This closes the integration half** of
+`E-TOKEN-BPE-CAN-FIT-NOT-YET-BUY-1` (#1012). That verdict measured that BPE FITS
+the `6×(8:8)` geometry and refused to buy a carrier. It did not ask whether one
+tokenization can SERVE several consumers at once. It can.
+**Confidence:** High for what is measured at these two corpus scales; the 8-bit
+lane saturated at 75 KB, so nothing here is a scale claim.
+
+**Headline.** ONE tokenization per span drove all three consumers and each added
+**zero** further tokenizations — not by discipline but by construction. Totals:
+313 source tokenizations for 308 spans plus 5 deliberate fixtures, and 1 QUERY
+tokenization on a deliberately separate counter (a query is different bytes;
+folding it into one number would make the claim a lie).
+
+**The two facts that make it adoptable, and neither was designed for this:**
+
+1. **DeepNSM-v2's library is already tokenizer-free.** `parse_to_spo(&[Tagged])`
+   consumes `(WordId, Pos)` pairs and touches no string; the
+   `split_whitespace`/`normalise` logic lives ONLY in `examples/bible_wave.rs`
+   and `examples/genre_shapes.rs`. The seam needed **no change to the crate**.
+   The `(WordId, Pos)` boundary is the shipped seam and nobody had used it as
+   one.
+2. **Tantivy structurally cannot own offsets.** Its indexer reads `Token::text`
+   and `Token::position`, uses `position_length` transiently, and reads
+   `offset_from`/`offset_to` NOWHERE outside its own tests. Offsets are consumed
+   only by snippet generation, which re-tokenizes the STORED text at query time.
+   An index cannot become the ABI here even by accident.
+
+**Measured, and the numbers are the point rather than the verdict:**
+
+| | KJV scene | Alice |
+|---|---|---|
+| bytes / spans / tokens | 1 126 / 8 / 354 | 75 514 / 300 / 37 149 |
+| compression | 3.18× | **2.03×** |
+| distinct ids used | 137 | **247 of 255** |
+| resident lane bytes | 832 (74 % of source) | 55 572 (74 %) |
+| receipt share of resident | 54 % | 30 % |
+| particles/span p50/p95/max | 4 / 8 / 8 | 8 / 30 / 43 |
+| tokens per lexical unit p50/max | 1 / 7 | 2 / 15 |
+| tokens straddling a word boundary | 30 | 58 |
+
+- **The 8-bit lane SATURATES.** 247 of 255 ids on 75 KB of ordinary English, with
+  compression already fallen from 3.18× to 2.03×. The canon's answer — the hi
+  byte of each `(8:8)` pair as a PAGE lane, two separate bytes, never a widened
+  `u16` — is **untested**. Until it is measured no scale claim about token BPE
+  should be made, and this supersedes any reading of #1012's 3.35× as a
+  corpus-independent figure.
+- **The resident lane is ~74 % of the source text, not a fraction of it**, and
+  at these span sizes **framing is 30–54 % of it**. A 56-byte receipt against
+  12-byte particles means the RECEIPT's column layout matters more than the
+  particle's. #1012 could not see this — it had no receipt.
+- **Cardinality is not 1:1 in either direction**, so a BPE↔`WordId` projection
+  is a real function, not a relabelling. Nothing in the seam assigns a `WordId`
+  to a BPE token; that would be a second vocabulary wearing DeepNSM's
+  coordinate system.
+- **Byte offsets are DERIVED**, by prefix sum over a per-id decoded-length
+  table. The receipt stores no offset column at all.
+
+**Four lance-graph-side gaps, each named rather than worked around:**
+
+1. **No shipped token continuation mechanism anywhere.** The nearest precedent
+   in SHAPE is `rail_geometry::RailCarving::AxisSlab { reg, cont: Option<usize> }`,
+   which chains one register to one continuation and caps at `RAIL_MAX_DEPTH = 24`
+   levels — **below the measured p50 of 4 particles**, so it does not fit. The
+   probe uses a contiguous run (`first_particle + particle_count + token_count`).
+   Stated honestly there are TWO lawful framings and the trade is exact:
+   `particle_count` alone bounds the run and a PAD scan inside that bound is
+   already exact BECAUSE PAD is reserved (cost: one vocabulary slot, which at a
+   255-cap that saturates is not free); or `token_count` costs 4 bytes and frees
+   the slot for a full 256-id alphabet. What is unlawful is inferring the end
+   from padding with no bound — measured, a lane-wide PAD scan overshoots
+   receipt 0 by 10 tokens straight into receipt 1.
+2. **`ValueTenant` has no token variant** (16 discriminants, none for text), so a
+   lawful resident lane must implement `SoaEnvelope` or land as a new tenant.
+   The probe's lane is a probe-local `Vec` and says so.
+3. **There is no callable part-of-speech surface, and the reason is a decision
+   already taken.** `coca_pos`/`archaic_pos`/`normalise` are byte-identical in
+   BOTH deepnsm-v2 examples, above a comment stating `deepnsm_v2::lexicon` was
+   DELETED after an audit found `lance-graph-planner`'s `insight_coca_read`
+   already grounds it. That grounding does not reach a lean consumer:
+   `insight_coca_read` is itself an **example binary**, in a crate carrying
+   `serde`/`serde_yml`/`tokio`/`ndarray`, and its master `lexicon.tsv` is absent
+   from this checkout. The probe restated the twenty-line tagger rather than
+   re-litigate the deletion — recorded so the next consumer has the evidence the
+   audit did not.
+4. **The semantic half is unexercised.** `cam96_codebook.bin` / `cam96_codes.bin`
+   are release assets, absent here, so palette256² DISTANCE never ran. Only the
+   lexical/grammar half was measured.
+
+**Polars: refuted, and the honest form is weaker than the question invited.** A
+sweep of nine checkouts found **zero** `polars` occurrences in any manifest or
+source; every `DataFrame` mention is prose. `paperless-rs` and `tesseract-rs`
+declare none of arrow/datafusion/lance/lancedb. There was nothing to remove. The
+structured-evidence path is likewise not tabular algebra:
+`lance-graph-arm-discovery` takes `Dataset { spec: FeatureSpec, rows:
+Vec<Vec<u32>> }` — category-index rows against a schema.
+
+**Method note, and it is the transferable part.** An independent vacuity audit of
+the finished probe found FIVE holes: a gate asserting byte counts but never span
+counts (the CRLF bug that collapsed 300 spans into 1 would have re-passed it), a
+threshold true by construction, an assertion about a type signature rather than
+behaviour, an unconditional prefix check, and an unexercised ASCII-vs-Unicode
+whitespace divergence. All five are fixed; four gained their own disable-runs;
+the fifth is bounded by a measured count of 0. Separately, TWO disable-runs were
+themselves wrong first — one relaxed a knob that does not bind on the fixture,
+one targeted a mechanism the gate did not actually rest on — and an early batch
+reported "no failure" six times in a row because the probe binary path was wrong
+and nothing ran. **A knob that does not bind is not a disable; a fixture's SHAPE
+is part of a test's coverage; and a null result is a claim about the apparatus
+until proven otherwise.**
+
+**⚠ SELF-CORRECTION, same session, after being pointed at `ogar-doc-ir`.** Two
+claims above were wrong and are corrected here rather than left standing:
+
+1. **The seam invented an identity that already existed.** The first cut minted
+   `source_id`/`span_id` integers. `ogar_doc_ir::DocIr` already answers all
+   three questions a tokenization receipt asks — `content_sha256` for WHICH
+   document, `(DocPage::number, Region::reading_order)` for WHICH span, and
+   `Region::text` for the span's canonical text. The probe was re-cut to read
+   them (`docir.rs`; gates `T-DOCIR` / `T-DOCIR-KEY` / `T-DOCIR-SPANS`, 41
+   total, 18 disable-runs). Note the crate's own docs CORRECT its plan's first
+   sketch on what that hash is: a **per-acquisition dedup key**, not a
+   cross-retina identity — which is exactly the right reading for a receipt,
+   because you tokenize bytes.
+2. **"The OCR boundary supplies no byte offsets" is RETIRED as a gap.** It
+   supplies no PAGE-wide offset and does not need to: a region owns its text,
+   so an offset is region-local, and `ogar-from-docv1::region_text` is where the
+   `leading_space`-aware join already happens. What remains is far smaller — a
+   sub-region span needs a non-zero `byte_from`, which the receipt already
+   carries and no producer emits.
+
+Also corrected: the `247 of 255` figure quoted above is the count of ids
+APPEARING in the lane, not the vocabulary size. Measured, the trained table is
+**full at 255/255** on Alice (and on the whole 170 KB file), and **180 of 255**
+on the KJV fixture — where, as #1016's own record of that fixture says, the
+CORPUS rather than the cap set it. The saturation conclusion holds and is
+stronger; the number was the wrong quantity.
+
+**Untouched by this.** HHTL is address geometry and BPE is tokenization —
+#1012's measured refutation of the merge tree as a radix prefix partition
+stands. Content never travels in classid: the contract id is a FIELD on the
+receipt, gated by a grep of the library's own non-comment source.
 ## 2026-08-23 — E-THE-SEVEN-OPCODE-PROJECTION-IS-NOT-X86-AND-THE-CHAIN-CARRIER-WINS-1 — four wave probes: the chain carrier confirmed, the vocabulary survives optimization, and the boundary that qualifies all of it
 
 **Status:** FINDING — [MEASURED] × 4 (`PROBE-R2IL-OPTIMIZATION-TRANSFER-1` 5/5,

.claude/board/LATEST_STATE.md

@@ -1,3 +1,32 @@
+## 2026-08-23 — #1017 OPEN — the token seam: #1012's integration half answered, and the 8-bit lane's ceiling found
+
+- **What exists now:** `PROBE-TOKEN-SEAM-1` (37 gates, 13 disable-runs) in
+  `AdaWorldAPI/paperless-rs crates/paperless-token`, with
+  `docs/TOKEN-SEAM-ARCHITECTURE.md` as its bounded architecture. It answers the
+  question #1012 left open: ONE versioned BPE tokenization of a span drives
+  Tantivy, DeepNSM-v2 and a forward-prediction input surface simultaneously,
+  each consumer BORROWING, none re-tokenizing. Neither Tantivy nor DeepNSM-v2
+  needed a line changed.
+- **Standing laws banked:** ONE SOURCE SPAN → ONE TOKENIZATION RECEIPT;
+  TOKENIZE ONCE, PROJECT MANY TIMES; AN INDEX MAY ACCELERATE THE ABI, IT MUST
+  NEVER BECOME THE ABI; BPE SEQUENCE IDENTITY IS NOT A SEMANTIC WORD
+  COORDINATE; and (method) A KNOB THAT DOES NOT BIND IS NOT A DISABLE — a
+  fixture's SHAPE is part of a test's coverage.
+- **The number that bounds prior work:** the 8-bit id lane saturates at 75 KB
+  (247/255 ids, compression 3.18× → 2.03×). #1012's 3.35× is a 1 KB figure and
+  must not be read as corpus-independent. The hi-byte PAGE lane is the next
+  probe and no scale claim survives without it.
+- **Open, in order:** (1) the paged vocabulary; (2) a real retina — this probe
+  builds its `DocIr` from text, so the next one should take
+  `ogar-from-docv1` on an actual scan and a `spider_doc_ir` crawl of the same
+  content and check both present the same span-population shape;
+  (3) a lawful resident lane (`SoaEnvelope` or a new `ValueTenant`, designed
+  against the measured 30–54 % framing overhead); (4) a real forward arm —
+  the seam supplies the input, but which representation a trained model prefers
+  is untested; (5) a structured-evidence corpus to exercise the parallel typed
+  path (`arm-discovery`'s `FeatureSpec` + category-index rows) and the shared
+  span identity where the two meet.
+
 ## 2026-08-23 — #1006..#1014 MERGED — the belief-ABI arc: Step 1 audit, root-law probes, Step 2 ruling request, frontier Phases 1+2, the real-episode measurement
 
 Nine PRs (#1010/#1011 landing stacked via #1009's merge) closed the arc:

.claude/board/PR_ARC_INVENTORY.md

@@ -1,3 +1,42 @@
+## 2026-08-23 — lance-graph #1017 (OPEN) — the integration half of #1012: one receipt, three borrowed consumers
+
+- **Added:** `E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1` — the board record of
+  `PROBE-TOKEN-SEAM-1` (37 gates, 13 disable-runs red-then-green; probe code in
+  `AdaWorldAPI/paperless-rs crates/paperless-token` + `docs/TOKEN-SEAM-ARCHITECTURE.md`,
+  which lives there because it needs a Tantivy dep this workspace does not carry).
+  ONE tokenization per span drove Tantivy, DeepNSM-v2 and a forward-prediction
+  surface; each added ZERO further tokenizations (313 source for 308 spans + 5
+  fixtures; 1 query on a separate counter).
+- **Locked:** DeepNSM-v2's library is ALREADY the seam — `parse_to_spo(&[Tagged])`
+  takes `(WordId, Pos)` and no string, so the crate needed no change; a BPE token
+  is never assigned a `WordId` (different id spaces, cardinality measured
+  non-1:1 in both directions); byte offsets are DERIVED by prefix sum over a
+  per-id length table, so a receipt stores no offset column; Tantivy cannot own
+  offsets (its indexer never reads them).
+- **Measured, and it bounds #1012's headline:** the 8-bit lane SATURATES — 247
+  of 255 ids on 75 KB, compression 3.18× → **2.03×**. The resident lane is 74 %
+  of source and framing is 30–54 % of THAT (56-byte receipt vs 12-byte
+  particles), so the receipt's layout outranks the particle's.
+- **Self-corrected in-session:** the first cut minted `source_id`/`span_id`;
+  re-cut onto `ogar_doc_ir::DocIr` (`content_sha256` + `(page, reading_order)` +
+  `Region::text`), so the receipt mints nothing. That RETIRES the
+  "no byte offsets at the OCR boundary" gap — offsets are region-local — and
+  corrects `247 of 255` (ids appearing in the lane) to a table that is FULL at
+  255/255 on Alice and 180/255 on the KJV fixture.
+- **Deferred / named:** no shipped token continuation mechanism (the
+  `RailCarving::AxisSlab` precedent caps at 24 levels, under the measured p50 of
+  4 particles); `ValueTenant` has no token variant; no callable PoS surface —
+  `deepnsm_v2::lexicon` was deliberately deleted and the `insight_coca_read`
+  grounding cited for it is an example binary outside a lean consumer's
+  dependency barrier; cam96 codebook/codes ABSENT so the semantic half never
+  ran; the hi-byte PAGE lane untested, which is the next probe.
+- **Refuted:** Polars in the online path — zero occurrences across nine
+  checkouts; `paperless-rs`/`tesseract-rs` declare no arrow/datafusion/lance/lancedb
+  at all. There was nothing to remove.
+- **Docs:** `E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1`; LATEST_STATE updated.
+- **Confidence:** High for the two measured corpora; no scale claim — the
+  vocabulary was full at 75 KB.
+
 ## 2026-08-23 — GAP FILLED — arc rows for #976..#1005 (consolidated by the orchestrator from a wave scribe's primary-source reconstruction)
 
 The gap marker recorded below this block is now DISCHARGED for #977..#1005.