Feature 007: the vacancy transform — field without location - #8
Merged
Conversation
Feature 006 shipped the closed lexicon; the source document argues that is only half the instrument. The other half is the vacancy transform -- Carroll's condition (full syntactic scaffolding, vacant lexical content) manufactured at a controlled rate. The contract is written before any code because feature 006 taught us why: one missing sentence about how a corpus becomes a token stream had the two stacks training on different data for a day. Eleven deliberate departures from the source are listed with reasons. Four of them (map built lazily, order-dependent give-up path, RNG-based seam fix, injectivity assumed) are corrections to bugs that break stability -- a property the source claims for itself. One (top64/2**64) would only ever have shown up across our two stacks, because that quotient is not exactly representable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract claimed the source's `top64 / 2**64` could land on different doubles in Python and JavaScript. I tested it: it does not. CPython's int true-division is correctly rounded, and JS's Number(bigint)/2**64 rounds once and then divides by a power of two, which is exact -- so both produce the same double. Checked on 200,006 values: 200,000 random 64-bit integers plus hand-picked ties at the rounding boundary. Every one matched bit for bit. The `>> 11` stays, for the reason that survives the measurement: a 53-bit integer over 2**53 is exactly representable, so the equality is structural rather than resting on the rounding argument above -- an argument that stops holding if someone later assembles the value from two 32-bit halves. Implementation is unchanged; only the justification was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Vacating content words changes three things at once: the forms become unknown, they fragment into more subword tokens, and the passage stops making sense. Only the first is "location". A caveat cannot separate them; a control can. The swap control mints a real, frequency-rank-matched English word instead of a nonce form -- same eligibility, same u(stem) < p decision, same injectivity guarantee. The passage is equally nonsensical but every form is a known word with ordinary tokenization. So: nll(swap) - nll(english) = the cost of wrong content nll(nonce) - nll(swap) = the cost of unknown form The UI must report that decomposition and never nll(nonce) - nll(english) alone. The residual tokenization component is not separable without a tokenizer-level control, and we say so rather than calling the remainder location. The control checks itself: swap must satisfy the invariance theorem exactly as nonce does, because the tiny model is blind to both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both would have broken the invariance theorem, and both survived my own review of the contract. Writing the implementation is what found them. 1. Case. The suffix was sliced case-preserved and then used in the seam test, so `gums` -> `flels` while `GUMS` -> `FLESS`: suffix[0] was `s` in one and `S` in the other, the seam fired in one and not the other, and one source type acquired two surface forms. The tokenizer lowercases, so those are two different types. The contract now requires the whole transform to commute with lowercasing, and states that as a normative invariant with a test. 2. Injectivity at intermediate p. The check ran at p=1 and over bare nonces. At full vacancy nothing can collide with a surviving English word, so the check saw nothing -- but at seed 7, `hang` mints `wak`, and `hanged` -> `waked` collides with the English `waked` at p=0.25 and p=0.5, where one is vacated and the other is not. The check is now over assembled surface forms with a p-independent condition, so it holds at every p simultaneously. Also resolved: a self-contradictory sentence in the mint loop (the reduced coda set is unreachable, as in the source -- do not "fix" it, it would change every multi-syllable nonce); the ambiguous stressTableCoverage, now a three-way split of table/minted/rule that sums to 1 and distinguishes stress we verified from stress we asserted; and four details the implementation had to invent, now pinned so the TypeScript side cannot diverge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two independent implementations both flagged that CODAS has 46 entries, not the 49 I wrote, and both correctly took "copy verbatim" over my tally -- so nothing diverged. The contract now says 46 and states plainly that the source lists are normative and the counts in this document are commentary: a nonce is a function of the strings and their indices, never of a number in a spec. The two stacks also reported different type counts (1922 vs 1665) while agreeing on tokensVacated to the token (8202). That is a definitional gap in this section, not a disagreement about the transform, so the definitions are now pinned -- typesTotal/typesEligible/typesVacated over the domain, stemsTotal/stemsVacated over the map, tokens over the corpus stream -- and stemsTotal/stemsVacated are added to the reported statistics. The identity that exposed it is now an assertion: at p=1 every eligible stem vacates, because u is in [0,1) by construction, so stemsVacated == stemsTotal and typesVacated == typesEligible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Python implementation surfaced a contradiction between the mint loop's give-up thresholds (400/800/1200) and the re-mint base salt (1000*round + ...). Read as absolute salt, a re-mint starting at 1001 would begin with every quality check already relaxed -- so the replacement nonce would not be prosody-matched -- and a second round at 2001 would exceed the give-up bound and raise, contradicting "raise after 8 rounds". A mint call now carries a base salt and an attempt counter; the byte stream is keyed on their sum, and the thresholds are on the attempt counter. A re-mint is therefore held to exactly the same standard as an original mint, which is what keeps the seed-7 replacement (hang -> smeeg) monosyllabic like the word it replaces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Implements contract sections 1-7 and 10. The map is built once over the whole type set in canonical order, so nesting and stability are structural rather than hoped for -- the source builds it lazily while rewriting, which makes a nonce depend on p and breaks the stability its own p-sweep needs. Injectivity is verified over assembled surface forms under a p-independent condition, not assumed. On the real corpus that costs exactly one re-mint, at seed 7: `hang` may not mint `wak`, because `hanged` would then collide with the surviving English `waked` at p=0.25 and p=0.5. The whole transform commutes with lowercasing, because the tokenizer lowercases and a case-sensitive step would give one type two surface forms. Prosody is reported as a three-way split -- table 5.1% of tokens before, minted 40.4% after, rule the rest -- which distinguishes stress we declared from stress the spelling heuristic guessed. No number from the source document is transcribed; these are measured on Mother Goose. SC-703 verified across 120 cases (2 seeds x 2 prosody settings x 6 budgets x 5 values of p): identical token id streams every time. 54 tests, ruff and black clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by running the module myself rather than reading the report: the contract said the domain is "corpus types union the budget's word list", which can be read as the ACTIVE budget. Under that reading, switching budgets in the panel would rebuild the map and re-mint the corpus in front of the reader -- while the panel is busy demonstrating that nonces are stable. The domain is now the full Dolch list, always. A frequency budget needs no special case, since its words are corpus types by construction. Measured, and now required as a test rather than trusted: the map is already identical across all five Dolch domains (gum -> thrern, hang -> smeeg at seed 7 for every one), because minting is keyed on (seed, stem) and the extra words provoke no new collisions. A future change to `avoid` or to the canonical order could break that silently, so it gets an assertion. Also verified independently at this commit: SC-703 holds on 120 cases, and the case-commuting invariant holds over 6,633 variants (every corpus type upper-cased, capitalised and lower-cased). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two stacks disagreed on type counts twice, both times because "types" is ambiguous between the corpus (2,211 types of Mother Goose) and the domain (2,233 = corpus plus the full Dolch list). They agreed on tokensVacated to the token both times, so it was never a disagreement about the transform -- only about what was being counted. An unprefixed types* is now forbidden. The panel shows corpusTypes*, because the 22 domain-only words are in the budget but never appear in the text and counting words the reader cannot see inflates the vacancy rate they are shown. domainTypes* stays as the diagnostic, since it is what governs the map. The TypeScript side was right and Python was wrong -- but the fault was this document, which said "domain" in one place while giving a definition that could be read either way. It also refused to silently reconcile its numbers to Python's when asked, which is why the cause was found in one round instead of being buried in a golden fixture. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two design points worth stating before anyone builds them. The nesting ribbon is the panel's real job. Nesting and stability are what make a p-sweep interpretable, and they are exactly the two properties the source's implementation breaks -- so the panel has to make them visible, not assert them in prose. Eight stems, five values of p, and the reader can see that a minted form never reverts and never changes. The invariance demonstration is split in two because the theorem is free to check and a training run is not: the token id streams are compared on every control change and the verdict is live (in a condition that breaks the theorem it must show the streams actually differing, never a hard-coded tick), while the training comparison is a button. And the framing constraint: the headline result is an exact zero, which a panel can easily make look like a broken chart. It says plainly that this IS the finding -- all of a word's meaning is field, none is form -- and points at the Architecture Explorer for the number that is not zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…is small The 2x2 now has numbers, from 6x250-word passages x 4 conditions x 2 models, with ONNX fp32 verified equal to torch at 5.3e-4 nats: nonce - english ~ 0.92-1.03 nats nonce - swap ~ 0.06-0.21 nats So 80-90% of the damage is wrong CONTENT and only 10-20% is unknown FORM. With the tiny arm's exact zero, that is the whole 2x2: a word's form is worth nothing to a model with no locations, and 10-20% of a nat to one that has them. It is the doc's T4 prediction -- field >> location -- holding on a model class it did not consider. The consequence for shipping is awkward and worth stating plainly. The effect I care about is small, and the app ships quantized ONNX: - pooled deltas survive q8 (|delta_q8 - delta_fp32| <= 0.054 nats) - per-passage deltas do not (worst 0.65 nats = 115% of that passage's delta) - nonce - swap is destroyed: 14-23% error pooled, sign flips in 1 of 6 passages - the error is not a removable offset -- q8 compresses extreme surprisal, and 2.7% of preserved tokens carry >5 nats of error, all on line-initial function words, which is exactly what this measurement probes And q4f16, the dtype the app tries FIRST, could not be measured outside a browser at all, so the deployed default path has no error bar yet. Measuring it in a real browser now. Policy: report a number only where a bound was measured for the dtype actually run; otherwise refuse and name the full stack. A stated uncertainty that was never measured is a fabricated error bar, which is worse than no number. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Not transcribed from the Python module -- written from the contract alone, so the two could disagree. They did, six times, and every disagreement was a defect in the contract rather than in either implementation. Two of them would have broken the invariance theorem silently. This side found three of the six: CODAS documented as 49 entries when the source has 46 (it copied the source verbatim and flagged the tally, which is why nothing diverged), the injectivity check being valid only at p=1, and the counting scope that this commit renames. On the last one it was asked to reconcile its numbers against Python's and told to stop and report instead if they disagreed. It did, and named the exact cause -- domain scope vs corpus scope, the 22 Dolch words absent from Mother Goose -- rather than quietly adopting the other stack's numbers. Both scopes are now reported under names that carry their scope, and corpusTypes* matches Python at every p and both seeds. Includes a one-line declaration fix in geoNodeEnv.d.ts: a namespace import of node:path makes the narrow ambient declaration win over @types/node for the whole program, exposing a missing `dirname`. Verified in a clean worktree that no error exists on the branch itself. 44 vacancy tests; 297 frontend tests; svelte-check 0 errors 0 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduces the TypeScript stack's domain-scope sequences exactly on the first run, and the domain change altered no measured number -- the fixture already used corpus types union the full Dolch list, which is now what the code says rather than what the caller had to remember. New tests pin the three things that cost a round trip each: that a re-mint is held to the same quality bar as an original mint (by forbidding a candidate at base salt S and showing it reproduces a fresh call at S+1); that the domain is never the active budget, with the map asserted byte-identical across all five Dolch domains; and that the two counting scopes differ by exactly the 22 Dolch words absent from Mother Goose. The field-name test now also asserts that no key starts with `types`, so the ambiguity that produced two rounds of reconciliation cannot come back. 59 vacancy tests, 208 unit tests, ruff and black clean, SC-703 120/120. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by running both stacks side by side rather than by reading either. Both implementations gave `avoid` a default of empty and left it to the caller to pass the corpus type set. Both agreed with each other, so no parity test could catch it -- but the map is then a function of what the caller remembered. Measured at seed 0: the same corpus and seed give remintRounds 0 with the domain passed and 1 without, with different nonces either way. Both maps are valid, which is exactly the problem. The panel passing it and the golden fixture not would be a silent divergence with no failing test. Condition B already requires that no surface form equal any domain type, so avoiding the domain at mint time is not an extra policy -- only the cheaper way to reach the same fixed point. Making it implicit costs nothing and makes the map a pure function of (domain, seed, matchProsody). Also pinned: both stacks expose a vacancyDomain helper (Python had one, TypeScript did not -- an asymmetry that ends with two call sites building the domain two ways), and Python must reject a bare str, since Iterable[str] accepts one and iterates it into a domain of single letters. I hit that myself in five seconds of use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reported as "the model saving/loading mechanism doesn't work". The Lexicon Lab
is fine in both modes -- train/save/load/regenerate is byte-identical for tied
and untied models, both budget sources, and hand-edited weights, and every file
the browser writes round-trips through POST/GET /api/lex/model unchanged. The
Geometry Lab is fine too, within one page load.
The defect is the static build, across a reload, for a model that has a
vocabulary of its OWN -- anything trained from scratch, or loaded from a file:
1. train from scratch on your text; Save writes your word list.
2. reload. The token is in sessionStorage and the weight set is restored, so
the model is still there.
3. Save again -> the SAME weights, under Alice in Wonderland's word list.
Silent, and unrejectable: vocab_sha256 is computed over the list that was
written, so the file verifies on both sides. It is exactly the corruption the
three digests exist to prevent -- real weights beside a substituted vocabulary
-- committed by the writer, where no digest can catch it. The same substitution
drove the sphere's labels and traces after the reload.
Cause: ExportedWeightSet, the persisted payload, carried weights, sources and
setSource but not the vocabulary, so tokenizerFor() fell back to the canonical
tokenizer and exportBundle hashed whatever it got. That fallback is right for
edited and fine-tuned sets, which really do keep the canonical words, and only
wrong for scratch and imported ones. The full stack was never affected: Python
has always stored the word list beside the weights (save_weight_set(...,
vocab_json=), load_weight_set_vocab).
So the word list now travels with the weights, the same way. A payload that
lacks one it needs is REFUSED rather than restored half-right, which lets
restorePersistedSets drop it and the evicted-token self-heal reset visibly; and
exportBundle refuses to write a file whose vocabulary it cannot vouch for, so
the silent path cannot come back. No integrity check was relaxed.
Both new tests fail before this commit and pass after:
* geoEngine.test.ts -- persist a loaded model with a distinct word list,
restore into a fresh engine, and require the re-exported bundle to be
identical; a payload stripped of its word list must be refused outright.
* static.spec.ts -- the reproduction above, in the real built static site:
train on the committed corpus, save, reload, save, compare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
buildVacancyMap(types, params) no longer takes `avoid`; the forbidden set is seeded with the domain itself, so the map is a pure function of (domain, seed, matchProsody) rather than of what the caller remembered to pass. This stack's numbers did not move, and it established why rather than assuming: it had been passing the corpus types as `avoid` since its first version, so it was already in the "domain passed" case. Making it implicit adds only the 22 Dolch-only words to the forbidden set, and no nonce in either map hits one of those 22, so the map is byte-identical to before. New vacancyDomain(types) applies the union rule and is used at every call site. It rejects a bare string: TypeScript has the same character-iteration trap as Python, where an Iterable<string> happily accepts a text and yields a domain of single letters. Four new tests, the useful one being that the map is byte-identical through two different call paths -- the helper, versus a reversed, duplicated, upper-cased, budget-first version of the same set -- which is what would have caught the avoid divergence. 48 vacancy tests; 301 frontend tests; svelte-check 0 errors 0 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract said the map is "identical across all five Dolch domains". That was measured when `avoid` was a caller-passed corpus type set, independent of the domain. Now the domain IS the forbidden set, so a smaller domain forbids less and mints differently: below `full`, exactly one stem moves -- jam -> floor instead of scirmp, because `floor` is a full-list Dolch word that never occurs in Mother Goose and so is only forbidden when the full list is present. This strengthens the rule rather than weakening it. It is now a reason the domain must be the full list, not a coincidence that it may be. The test now asserts the property that actually matters -- the map does not move when the active budget changes -- instead of the stronger claim that happened to hold before. Caught by the Python implementation re-measuring after the change rather than assuming the old number still held. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rables Moves onto the same map TypeScript already had: seed 0 remintRounds 0 with gum -> fles and hang -> pruch, seed 7 remintRounds 1 with thrern and smeeg. The two stacks now agree on the nonces themselves, not merely on counts. Guards added where an Iterable[str] silently accepts a text and iterates it character by character -- vacancy_domain, build_vacancy_map, map_vocab_words, and VacancyParams(keep=...), which had the same trap with keep="little". I hit this myself within seconds of using the module: the domain became single letters and the failure surfaced much later as an unrelated-looking error. 61 vacancy tests, 210 unit tests, ruff and black clean, SC-703 120/120. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ran both stacks over the real corpus and diffed every stem->nonce pair across four map configurations plus three partial-p vacated corpora: 6,720 pairs, 7 configurations, 0 mismatches, and all 7 vacated-corpus sha256 digests identical. remintRounds agrees too (seed 0: 0, seed 7: 1). That is the result the golden fixture will formalise; this commit records that it was established by driving both implementations, not by comparing two reports that each said "green". Also pins the last naming asymmetry: VacancyMap's stem->nonce field is `mapping` in both stacks. TypeScript called it `map`. That is the third such asymmetry this feature produced, after the missing vacancyDomain helper and the avoid default, and each one cost a round for anything driving both stacks at once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third and last naming asymmetry of this feature. Cosmetic, but each of these cost a round for anything driving both stacks at once, and the golden fixture and the panel are both about to do exactly that. Parity re-verified after the rename: 6,720 stem->nonce pairs and 7 vacated-corpus digests, zero mismatches. 48 vacancy tests; svelte-check 0 errors 0 warnings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were found because the feature forced us to drive the real app against a known-good reference, and neither is a 007 bug. The Geometry Lab one is fixed. The q4f16 one is live: the dtype the app tries first builds a session successfully on WebGPU and then returns degenerate output, and the fallback only fires on a thrown exception, so the deployed Architecture Explorer is showing wrong probabilities on any shader-f16 machine. The lesson worth keeping is that both produce PLAUSIBLE wrong answers. Unit tests pass, nothing throws, the UI looks normal. What caught them was running the real thing and comparing against a reference -- and in the q4f16 case, the reason it survived this long is that headless Chromium exposes no WebGPU adapter, so the e2e suite has only ever exercised wasm/q8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were invisible to my own cross-language check, which only exercised
consistent=true / revealAfter=0. Each stack was right about a different one.
1. consistent=false prosody. Section 5.8 pinned the mint key `stem#idx` but not
which string the stress pattern comes from. One stack passed the key into the
minter, so `little` got stress("little#0") = "10" instead of
stress("little") = "100", and `Little` minted as Wrerken rather than
Wrerkenle. Section 7.1 says the nonce carries THE STEM's syllable count and
stress, so the key feeds the byte stream and the uniqueness check only and
must never reach the prosody lookup.
2. corpusTypesVacated under revealAfter > 0. One stack measured the two texts
(665), the other asked whether the stem was in the vacated set (1337) --
over-reporting 2x, because a type whose every occurrence falls inside the
reveal window is still listed in the map. Measuring the texts is the reading
that matches what the number claims to the reader, so that is now the
definition. At revealAfter = 0 the two coincide, which is why only a control
condition could expose it.
The fixture pinned both readings with the cause and the status, and refuses to
regenerate once either is fixed -- so neither can be quietly forgotten.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… data 181 KB, 18 cases, byte-identical across three regenerations. Pins u(stem) as exact float64, the full 1680-pair map at two seeds, the vacated corpus head and sha256 at seven values of p, all 23 statistics per case, nesting as explicit stem sets, stability as each stem's surface form at every p, and the token-id stream digest under the mapped vocabulary -- identical at every p, both seeds, both prosody settings, which is SC-703 pinned as data rather than as an assertion in one language. 64 tests, 198 float comparisons, worst delta 0. It earned its keep immediately by finding two defects in the CONTROL conditions, which my own cross-language check had missed because it only exercised consistent=true with revealAfter=0. Each stack was right about one of them. Both are pinned here with both readings, the cause, and the status, and the exporter refuses to regenerate once either is fixed -- so a green suite cannot quietly outlive them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The p slider, the seed, the three control conditions, the live corpus view coloured from the real map, the nesting ribbon, and the statistics -- all from vacancyStats, no number transcribed from the source document. The nesting ribbon is the part that matters: eight stems by rank of u, across five values of p, so a reader can SEE that a minted form never reverts and never changes. Those are the two properties that make a p-sweep interpretable and exactly the two the source's implementation breaks; asserting them in prose would have been worthless. The invariance demonstration runs in two tiers because the theorem is free to check and training is not: the token id streams are compared live on every control change (19,071 ids), and a button trains at two values of p. On screen: max |delta loss| = 0, final loss 4.440053 in both runs. TrainPanel gained vocabWords, passed when p > 0 -- without it the worker rebuilds a Dolch list from a corpus that no longer contains those words. The mint: "swap" control is rendered DISABLED with a note naming the contract section, because neither stack implements it yet. Aliasing it to "nonce" would have looked like it worked. svelte-check 0 errors 0 warnings; 374 unit tests; 16 e2e. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shipped corpus carries 618 token-producing lines of front matter -- title page, list of rhymes, index of first lines -- before LITTLE BO-PEEP starts the verse at line 619. Landing there made the transform look like it rewrites an index, which is the least interesting thing it does. Pinned rather than detected, and the reason is worth recording: every general rule I tried picks the wrong boundary. The index of first lines has verse-length lines (median 6 tokens), so a line-length heuristic stops at line 320, still inside the front matter. Separating an index from a stanza needs to know the book. The corpus is committed and digest-verified, so a constant is the honest way to say so, and the e2e test asserts the default page really opens on Bo-Peep. Pasted text and HuggingFace datasets open at line 1. Includes a reactivity bug the new test caught: the pager set `userPaged` before reading `win`, which made `win` fall straight back to `windowIndex` -- still 0 while the default was in force -- so the first click jumped to line 1 instead of stepping back one page. Capture, then assign. 9 e2e tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by Python while fixing the prosody defect: at seed 7, p=1, the stem `tak` minted the nonce `tak`, so `Taking` -> `Taking`. A token silently failed to vacate, and corpusTypesVacated came out 1921 against the consistent path's 1922. Condition B was enforced when building the map and not on the per-occurrence path. Section 7.1 says this control has no STABILITY property -- that is about a nonce being reused across occurrences, and it does not license a word quietly surviving the transform. A control whose vacancy rate is not the stated rate is not a control; the whole point of `inconsistent` is to hold the rate fixed while destroying the identity. So a per-occurrence nonce must equal neither any domain type nor the stem it replaces, under the same re-mint loop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s garbage On any machine whose browser exposes a WebGPU adapter with shader-f16 — any recent Mac — the Architecture Explorer's first-choice dtype builds a session successfully and then returns logits that carry no information about the input. Nothing throws, so the exception-only fallback ladder never fired, and the deployed site has been showing per-token probabilities, traces and generations computed from those logits. Re-measured here in a real browser on the real Apple Metal-3 adapter, L-infinity gap between the first and last row of one teacher-forced pass, plus a greedy continuation: gpt2-ONNX webgpu/q4f16 0.000 ",,,,,,,,,," SmolLM2-135M-Instruct-ONNX webgpu/q4f16 0.000 all logits exactly 0, empty SmolLM2-360M-Instruct-ONNX webgpu/q4f16 0.000 all logits exactly 0, empty Qwen2.5-0.5B-Instruct webgpu/q4f16 16.469 not degenerate, but worse gpt2-ONNX webgpu/q8 91.989 " Berlin. The capital of ..." SmolLM2-135M-Instruct-ONNX webgpu/q8 36.968 " Berlin.\n\nThe capital of ..." Qwen2.5-0.5B-Instruct webgpu/q8 20.265 " Berlin. What is ..." Three of the four curated models are destroyed by it. The primary fix is therefore configuration, not detection: the ladder is now webgpu/q8 -> wasm/q8. q4 is correct too (SmolLM2-135M: 35.663, correct continuation) but is not the smaller download it was assumed to be — in every curated repo model_q4.onnx is LARGER than model_quantized.onnx (gpt2 498 vs 280 MB; SmolLM2-135M 181 vs 136; SmolLM2-360M 386 vs 363; Qwen2.5-0.5B 786 vs 512). Both rungs now read the same file, so a rejected WebGPU rung costs no second download. Because the failure is silent, "the session constructed" is not evidence that the model works. logitsSanity.ts states ONE invariant, in the spirit of the Geometry Lab's training gates: a causal LM's output must depend on its input, so the L-infinity gap between the first and last next-token distribution of a fixed 12-token probe must exceed 1e-3. All-identical rows and all-zero logits are the same failure, not two special cases. Every session is checked at load, before any number is shown; a rejected rung falls to the next and the badge NAMES the device/dtype actually in use, marking "fallback" when a rung was rejected. The unforgivable part of this defect was invisibility, not the fallback. Coverage, which is why it shipped: plain headless Chromium has no WebGPU adapter at all, so the whole e2e suite only ever exercised wasm/q8. tests/e2e/webgpu.spec.ts plus a `webgpu` Playwright project closes that on hardware — on macOS the flag that matters is --use-angle=metal, since --enable-unsafe-webgpu alone still hands back google/swiftshader. Verified to FAIL against the pre-fix code (badge "webgpu q4f16"; 2 distinct top-5 lists across 64 generated positions) and pass after (64/64 distinct). NAMED RESIDUAL GAP: GitHub-hosted runners have no GPU, so that test SKIPS in CI with a loud reason rather than passing vacuously on WASM — a green CI run does not mean the WebGPU path was checked. What CI does always verify: tests/unit/logitsSanity.test.ts (the invariant and the dtype ladder) and tests/e2e/static.spec.ts (a real session through the same gate on the WASM rung). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inconsistent control could leave a word un-vacated: at seed 7 the stem `tak` minted the nonce `tak`, so `Taking` -> `Taking` and corpusTypesVacated came out 1921 against the consistent path's 1922. Condition B was enforced when building the map but not per occurrence. A control whose vacancy rate is not the stated rate is not a control, so both stacks now forbid a per-occurrence nonce from equalling the stem it replaces as well as any domain type. Both seeds now give 1922 / 8202, matching the consistent path, with byte-identical vacated-corpus digests across the two stacks. The fixture is regenerated over three landed fixes and its KNOWN_DIVERGENCES list is now empty. Both entries were removed rather than bypassed -- the exporter's refusal fired on control-inconsistent.vacatedSha256, which is the guard working. The guard was re-probed both ways afterwards: a synthetic outstanding divergence is still recorded, and a resolved one still raises. A new control-inconsistent-seed7 case pins the `tak` fix so it cannot regress. pytest 400; vitest 384; svelte-check 0/0; golden 209 float comparisons, worst delta 0; regenerated twice byte-identical; SC-703 re-verified 120/120. Reported and not changed: Section 5.8 lists `forbidden` as a normative VacancyMap field, but Python reconstructs it from domain plus mapping values rather than storing it, so it omits superseded re-mint nonces that TypeScript retains. Nothing observable diverges today -- every digest agrees -- but the two sets are not the same set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One stack stored the set; the other rebuilt it as domain plus mapping values, which silently drops every nonce a re-mint round replaced (`wak` at seed 7). Nothing observable diverges today -- every digest agrees -- but the two sets are genuinely different, and the per-occurrence path of the inconsistent control now draws against `forbidden`, so it is one unlucky hash away from mattering. Superseded nonces stay forbidden because they were rejected for a reason: reusing one can recreate the collision the re-mint resolved. Reported by the agent that fixed the `tak` defect, as a latent asymmetry rather than a failure -- which is the right way to surface something no test catches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Recording this because the method mattered more than any individual fix. The two stacks were written independently from the contract precisely so they could disagree, and every disagreement turned out to be a defect in the document rather than in either implementation. I caught none of them by re-reading. Four would have broken the invariance theorem silently. Three were found only because a stack was told to STOP and report rather than reconcile its numbers to the other -- a golden fixture built over a silent reconciliation would have cemented both stacks being consistently wrong, with a green suite on top. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Feature 007's FR-721/FR-722. The transform already existed in both stacks; this is the layer that lets the panel ask for it and gets the same answer either way. * `POST /api/lex/vacancy` (routes_lex.py) — the corpus-source and budget rules of /api/lex/coverage, plus contract §7.1's knobs, returning §10's statistics under their exact camelCase names, the resolved vocabulary, and which of §7.2's two rules produced it (mapped vs rebuilt). * `POST /api/lex/train` gains an optional `vacancy` object. Absent, byte for byte the endpoint it was. Present, the corpus is vacated server-side and read under §7.2's vocabulary — which under the theorem's conditions makes the losses BIT-IDENTICAL to the English run, asserted rather than described. * `staticClient/lex.ts` computes both live in the browser. The Lexicon Lab is browser-side in BOTH modes, so there was nothing here to refuse. An excerpt and a digest, not the corpus. The vacated text is ~86 kB and the panel re-runs this on every tick of the `p` slider; nothing needs it whole, since the panel shows a screenful (the source's own figure is 400 characters) and a caller that wants to TRAIN on it sends the same parameters to /api/lex/train. What an excerpt cannot do is prove which text it came from, so `vacated_sha256` pins all 86 kB in 64 characters — and that digest is also the single value the two stacks are compared on. The parity that matters: `tests/fixtures/vacancy-api-golden.json` is a transcript of the real route (scripts/export_vacancy_api_golden.py, real app, real corpus, no mocks). `test_api_lex.py::test_vacancy_matches_the_static_client_fixture` asserts the live route still returns it; `staticVacancy.test.ts` asserts the browser reproduces it field for field — statistics, vocabulary, coverage, preview and digest, across six cases including both control conditions and a frequency budget. One document, two stacks, no way for either to drift alone. Measured and asserted on the shipped corpus: domain 2233 types, 1680 stems, seed 0 `remintRounds` 0 and seed 7 `remintRounds` 1; at p=1, `corpusTypesVacated` 1922 of 1922 eligible (2211 total) and `tokensVacated` 8202. Nothing in lex/vacancy.py or lexEngine/vacancy.ts was touched: `VacancyParams` is built by keyword and the train cache key is read off the dataclass's own fields, so the `mint: "swap"` parameter landing in those modules costs one line here and cannot silently reuse a cache entry made before it existed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ommit) The frozen feature-002 contract is edited in its own commit, per the project rule, and the edit adds NOTHING to the endpoints it froze — no path, parameter, field, status code or error type below the new section changes meaning. What changed and why it was needed: * `api.md` gains an "Additive namespaces" table recording that 006 added `/api/lex/*` and 007 adds `POST /api/lex/vacancy` plus an optional `vacancy` object on `POST /api/lex/train`, with the reasoning for each. 006's namespace was already shipped and already documented in its own file, but nothing in the frozen document said so — "frozen" was quietly meaning "undocumented", and the next reader would have had to grep the routers to learn what else exists. * Why 007 needed a new endpoint rather than a flag on an existing one: the vacancy transform rewrites a CORPUS and every endpoint in the frozen file takes a model or a prompt. Folding it into `/api/lex/coverage` would have made that response mean two different things depending on a flag, which is the failure the freeze exists to prevent. `/api/lex/train` did gain a parameter, and it is optional with an absent-means-unchanged guarantee, because a vacated corpus must be tokenized under the vocabulary the transform assigns it and shipping the ~86 kB rewritten text back and forth to arrange that would have been the worse contract. * `api-lex.md` gains the full wire specification: request and response of `POST /api/lex/vacancy` (including why it returns an excerpt plus a sha256 rather than the corpus), the `vacancy` object on `/train` and its bit-identical loss guarantee under the mapped vocabulary, the error list, and the parity fixture both stacks are asserted against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SC-707a asserted that swap must satisfy the invariance theorem exactly as nonce
does, and offered that as the check that swap was implemented correctly. It is
false, and the reason is a theorem rather than a bug:
A map that is stable in p and whose images are DOMAIN TYPES is injective at
every p only if it is the identity.
T_p injective for all p forces sigma(V_p) = V_p; the V_p grow one stem family at
a time, so u(sigma(s)) = u(s), so sigma = id. Swap draws its replacements from
the domain by construction, so at intermediate p a swapped word collides with a
word not yet vacated -- measured, 191/246/190 colliding types at p = .25/.5/.75.
So the criterion is corrected, not the code weakened: swap satisfies the theorem
at p in {0,1} -- 48 of the 120 SC-703 cases -- and the other 72 are REFUSED with
a typed error citing the new section 5.2a. Nonce stays 120/120. An instrument
that declines the configurations it cannot support is sound; one that quietly
returns a non-injective map is not. This costs the pretrained arm nothing, since
it scores at full vacancy, where swap IS a bijection of the domain -- checked
rather than assumed.
Condition B is also restated properly: B is sufficient, not necessary. The exact
p-independent condition is A + B', and B implies B' vacuously, so nonce is
unchanged.
Also closes the `forbidden` asymmetry: Python now stores the set including every
superseded re-mint nonce, matching TypeScript. At seed 7, `wak` is in `forbidden`
but not in `mapping.values()` -- named in both stacks' tests. Every pre-existing
fixture case regenerated byte-identical, so nothing observable moved.
Both stacks agree on all four swap map digests and all six swap id-stream
digests. Backend 462; frontend 453; svelte-check 0/0; fixture byte-identical
across two regenerations.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes the doc's T4 2x2 with measured numbers. fp32 backend, 6 pooled corpus excerpts, p=1, differences PAIRED over preserved tokens (identical token counts in all three variants, so they are exactly additive): model wrong content unknown form total pairs gpt2 0.717 +- 0.054 0.273 +- 0.041 0.989 847 SmolLM2-135M 0.830 +- 0.052 0.160 +- 0.035 0.990 859 Qwen2.5-0.5B 0.990 +- 0.059 0.222 +- 0.040 1.212 823 So a word's FORM is 16-28% of the damage and its CONTENT is the rest. Beside it sits the tiny arm's exact 0: the same transform is worth nothing at all to a model with no locations, and a fifth of a nat to one that has them. That pair is the whole instrument. `nonce - english` is returned but flagged headline:false, because on its own it conflates the two. A recommendation inherited from the earlier quantization study did not survive contact with the shipping configuration. That study proposed stating +-0.1 nats; measured here on real static q8 with byte-identical tokenization, the cross-stack gap was 0.073 and 0.110 -- so the number would have been violated by its own first comparison. The stated uncertainty is +-0.2, about twice the largest gap actually observed, with the derivation in the constant's docstring. Static mode refuses, each with a typed error naming the full stack: nonce - swap (14% error measured on this very configuration), every absolute NLL, every per-passage row, any dtype outside q8, and any pooling under 700 preserved tokens. Also fixes default_passages, which was cutting excerpts from the book's alphabetical index of first lines -- the same front-matter trap the Lexicon Lab panel hit, now skipped by the measured boundary in both places. backend 462 pytest, ruff + black clean; frontend 453 vitest, svelte-check 0/0; e2e chromium 2, static 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Lexicon Lab's `mint` control was rendered disabled: `LexiconLab` never
passed `mint` into `vacParams` at all, so it could not have worked. It works
now, and the interesting part is how its one real constraint is presented.
`mint = "swap"` is injective only at p in {0, 1} — a theorem (contract §5.2a),
not a rough edge: a map stable in p whose images are domain types is injective
everywhere only if it is the identity. So the UI never clamps p, never falls
back to nonce, and never catches-and-replaces the typed error. `LexiconLab`
asks the engine and carries the refusal up as a string; the panel prints it
verbatim with the four exits offered as buttons, and with no vocabulary the
counters, the trainer and the invariance check have nothing to report — which
is the honest state of that configuration.
Beside the mint control the theorem is COUNTED rather than asserted: every
domain type is pushed through the real transform at the current p and the
distinct images counted. Measured on the shipped corpus at seed 0, swap loses
244 / 322 / 233 image slots at p = .25/.5/.75 and 0 at both endpoints; nonce
loses 0 everywhere. The `bijective` chip now reads "injective at p = 0, 1"
under swap instead of a bare tick.
Info tab gains #vacancy: the T4 2x2 with the vacancy cell marked, the
transform's definition, nesting and stability and the four properties the
source implementation claims and breaks, the invariance theorem with §7.4's
framing (the exact zero IS the finding), the swap decomposition with "cost of
unknown form" stated as an upper bound, the stress table's real status (61
rule-seeded entries, never human-checked, 5.1% of this corpus's tokens), and
what the static build refuses by name — plus the WebGPU/CI coverage gap.
#real, #limits and #refs updated.
Every number in that prose is pinned by five new tests in docs.spec.ts: the
counts come from a live POST /api/lex/vacancy, the swap collisions and the
stress-table size are read off the running panel, and the static build's
±0.2 nats and 700-token floor are read out of the constants that enforce them.
No number from the source document appears anywhere; its prosody figures are
its own, on a corpus we do not have.
Also fixes a stale comment in staticClient/arch.ts that still documented a
±0.1 nats quantization uncertainty its own constant had superseded with 0.2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third occurrence in two sessions. The export refused to publish because the Hub 429'd through all five retries and `main` never resolved to a commit sha -- which is the guard behind issue #5 working, not a bug. Re-running clears it, but the recurrence is the point. Two fixes worth doing: an HF_TOKEN repository secret (needs the owner; CI is unauthenticated and rate-limited hard), and pinning the curated models to explicit commit shas, which removes the main->sha resolution from the build path entirely and does not need the owner. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The secret was added to the repository but neither workflow passed it to the environment, so it was doing nothing. Both now export it at the workflow level. Verified rather than assumed: huggingface_hub 1.19.0's get_token() returns the environment value when HF_TOKEN is set and None when it is not, so both `huggingface_hub` and `transformers` pick it up with no code change. Optional by design -- forks and fork PRs get an empty string, which the Hub treats as anonymous, i.e. exactly today's behaviour with today's retries. Nothing breaks without it; it is only more likely to succeed. This matters most in the Pages job, which runs the real backend as a build tool and resolves every curated model's revision to a commit sha. On a 429 that resolution fails and export_static_assets.py REFUSES to publish rather than ship weight URLs pointing at a moving `main` (issue #5) -- so an unauthenticated rate limit does not degrade the deploy, it blocks it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the vacancy transform to the demo — the instrument
~/Desktop/TinyModelsDoc/tiny_models.texargues a tiny model is for, and the half feature 006 deferred as FR-624.Both arms of the doc's T4 2×2, as chosen: the tiny arm in the Lexicon Lab, the pretrained arm in the Architecture Explorer.
The result
A word's form is worth:
Even where a location exists, losing it costs about a fifth of what losing the content costs. That is the doc's T4 prediction — field ≫ location — measured on a model class it did not consider. The exact zero is what gives the other number a scale.
Two shipped defects fixed here, unrelated to the feature
Both were found because the feature forced us to drive the real app against a known-good reference. Both produced plausible wrong answers — nothing threw, no test failed.
d6e9d5d— the Geometry Lab saved weights under the wrong vocabulary. Static build, across a reload, for any model with its own word list: train → Save → reload → Save gave the same weights under Alice in Wonderland's vocabulary. Unrejectable, becausevocab_sha256was computed over the substituted list, so the file verified on both sides.c115ec6— q4f16 returns garbage on WebGPU. The dtype the app tried first. The session builds successfully and returns logits carrying no information about the input; three of four curated models are destroyed (gpt2 emits,,,,,,,, both SmolLM2 models return all-zero logits). The fallback only fired on a thrown exception, and nothing throws. The deployed site has been affected on anyshader-f16machine.Verification
toEqualsvelte-checkKnown gaps, stated rather than implied
shader-f16. The path is verified locally on a real Metal-3 adapter. CI always covers the non-degeneracy invariant, the dtype ladder, and a real session through the same gate on wasm. Documented in the test,ci.yml, and the spec.nonce − swap, every absolute NLL, per-passage rows, non-q8 dtypes, and pools under 700 tokens — each a typed error naming the full stack. q8's error on that difference is 14–23% of an effect whose true value is 0.06–0.21 nats, with sign flips.swapis injective only atp ∈ {0, 1}— a theorem (contract §5.2a), not a rough edge. The 72 unsupported SC-703 configurations refuse rather than returning a quietly non-injective map.Process note
The two stacks were written independently from a normative contract so they could disagree. Thirteen defects were found in the contract, none by re-reading it — four would have broken the invariance theorem silently. Three were found only because a stack was instructed to stop and report rather than reconcile to the other; a golden fixture built over a silent reconciliation would have cemented both stacks being consistently wrong under a green suite.
One of the retractions is my own: SC-707a asserted that
swapmust satisfy the invariance theorem asnoncedoes. It provably cannot, and the original criterion would have been "satisfied" by any implementation willing to skip the injectivity check for one strategy.Specs:
specs/007-vacancy-transform-field/{spec,architecture,ui}.md. Frozen HTTP contract updated additively in its own commit (1b4704b).🤖 Generated with Claude Code