Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# 2026-09-26 — DeepNSM-v2 keeps counted lexical evidence beside routing

**Status:** MEASURED · DONE — `crates/deepnsm-v2/src/lexical.rs`

## The loss point
`PaletteVocab::from_frequency_ranked` (`vocab.rs`) admits surface strings, first wins. The counts and PoS behind the ranking never reach v2 storage. `examples/bible_wave.rs::load_pos` repeats the loss for PoS: `word -> one Pos`, counts dropped.

## What landed
`LexicalEvidence`, built against an existing `PaletteVocab` and never changing it. One `WordId` owns N `LexicalReading { pos: PosCode, lemma: Option<LemmaRef>, form_count: Option<u64> }`. `LemmaEntry { source_key, lemma, pos, count }` is keyed by the source's `lemRank`. Queries: `surface_count`, `surface_pos_count`, `lemma_pos_count`, `lemma_count`. All counts are `Option<u64>`: an empty field is unknown, a literal `0` is zero. Conflicting lemma entries and duplicate readings are refused, not first-wins. Loader: `load_word_forms_csv` (exact header `lemRank,lemma,PoS,lemFreq,wordFreq,word`).

## Measured on the committed COCA `word_forms.csv`
11,460 rows → 11,456 readings + 4 empty-surface rows (reported), 0 unrouted when the vocab covers every surface, 5,050 lemma entries (= distinct `lemRank`). 1,165 surfaces carry more than one reading, e.g. `record` n 120,048 / v 13,014 under lemma totals 187,057 / 51,375.

## Open
- The trained Cam96 path (`bible_vocab.txt`) is surface-only: no counts, no PoS. KJV routing gets evidence only when a caller joins a counted source to it.
- `academic_20k.csv` (surface × PoS × COCA-All/Acad, no lemma) has no loader. It holds 3 same-(word, PoS) row pairs with different counts, so a loader must decide whether those are disjoint before summing. The builder refuses a second (word, PoS, no-lemma) reading as `DuplicateReading` until that is decided.
3 changes: 2 additions & 1 deletion .claude/board/entries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
opposite directions; the stranding this convention prevents shows up in
exactly one of them, never both.

174 entries, 2026-08-06 .. 2026-09-25.
175 entries, 2026-08-06 .. 2026-09-26.

| date | entry id | finding | file |
|---|---|---|---|
| 2026-09-26 | `deepnsm-v2-lexical-evidence-survives-routing` | | [2026-09-26-deepnsm-v2-lexical-evidence-survives-routing.md](2026-09-26-deepnsm-v2-lexical-evidence-survives-routing.md) |
| 2026-09-25 | `window-scheduling-and-two-level-ternlog` | | [2026-09-25-window-scheduling-and-two-level-ternlog.md](2026-09-25-window-scheduling-and-two-level-ternlog.md) |
| 2026-09-25 | `tile-knee-and-compiled-lowering` | | [2026-09-25-tile-knee-and-compiled-lowering.md](2026-09-25-tile-knee-and-compiled-lowering.md) |
| 2026-09-25 | `tern3-six-view-window` | | [2026-09-25-tern3-six-view-window.md](2026-09-25-tern3-six-view-window.md) |
Expand Down
9 changes: 9 additions & 0 deletions crates/deepnsm-v2/data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,12 @@ curl -L -o bible_vocab.txt "$BASE/bible_vocab.txt" # frequency-ranked vo
Provenance + held-out metrics: `../probes/README.md` §4 (producer scripts:
`../probes/{embed_bible_vocab,train_codebook}.py`). Loaded by
`deepnsm_v2::codebook::{load_cam96_space, load_cam96_codes}`.

## Lexical evidence (counts + PoS) — not in the Cam96 release

`bible_vocab.txt` carries surface forms only, in rank order: no counts, no
part of speech. Counted evidence is read separately by
`deepnsm_v2::lexical::load_word_forms_csv`, whose input is COCA
`word_forms.csv` (`lemRank,lemma,PoS,lemFreq,wordFreq,word`, committed at
`crates/deepnsm/word_frequency/`). It is stored beside the routing
`PaletteVocab`, never inside it and never inside the Cam96 codes.
Loading
Loading