Measure structural sharing after FP16/INT8 on trained n-gram tables

README.md

@@ -108,6 +108,7 @@ external data or GPU. It checks integration, not representative compression.
 | --- | --- |
 | [Actual FastText model](docs/fasttext_native_evidence_20261031/README.md) | 90,380 rows and word/sentence outputs byte-identical; whole bundle saves 0.46%, hot queries slower than dense; structural order has no added benefit here |
 | [Structural contribution controls](docs/structural_native_evidence_20261030/README.md) | Trained SGNS/GRU references measure DATrie and approximate joint sharing separately; exact mode finds no identical rows to merge |
+| [FP16/INT8 + structural controls](docs/quantized_structural_evidence_20261031/README.md) | Same trained tables retain additional approximate savings after quantization; 4-order GRU INT8 regresses; quality failures and fixed-default audit are separate |
 | [1.85M trained-row capacity](docs/cpu_capacity_evidence_20261030/README.md) / [sparse blocks](docs/sparse_blocks_evidence_20261030/README.md) | Smaller blocks lower native miss latency 16–27× relative to the previous native layout; raw mmap remains faster |
 | [TinyEngram/Qwen trained integration](docs/tinyengram_qwen_evidence_20261030/README.md) / [CUDA controls](docs/qwen_cuda_performance_20261030/README.md) | Preserved trained outputs and measured CPU/device costs; native lookup remains slower than dense offload on these traces |
 

docs/index.md

@@ -15,6 +15,7 @@
 
 - [Actual legacy FastText model and matched layout controls](fasttext_native_evidence_20261031/README.md)
 - [Original DATrie / mark-compact contribution ablations](structural_native_evidence_20261030/README.md)
+- [Same-trained-table FP16/INT8 and structural-sharing controls](quantized_structural_evidence_20261031/README.md)
 - [1.85M trained-row CPU capacity](cpu_capacity_evidence_20261030/README.md)
 - [Sparse-block native miss latency](sparse_blocks_evidence_20261030/README.md)
 - [Native query/cache controls](native_query_evidence_20261030/README.md)

docs/ngram_memory_research_plan.md

@@ -125,6 +125,7 @@ work.
 | Original-ID/byte-preserving storage and adapters | [Actual FastText outputs](fasttext_native_evidence_20261031/README.md), [trained TinyEngram/Qwen](tinyengram_qwen_evidence_20261030/README.md), Linux/macOS wheel CI | Frozen handles; reduction contracts differ; official Engram verification uses a demo, not trained weights |
 | Sparse native queries | [1.85M trained rows](cpu_capacity_evidence_20261030/README.md) and [smaller blocks](sparse_blocks_evidence_20261030/README.md) | Misses improved 16–27× over the prior layout; raw mmap remains faster |
 | DATrie / mark-compact | [Trained SGNS/GRU ablations](structural_native_evidence_20261030/README.md) | Exact key indexing helps these traces; no identical structural rows found; extra joint row savings are approximate |
+| Quantization + structural sharing | [18 trained tables, 486 controls](quantized_structural_evidence_20261031/README.md) | Additional savings remain after FP16/INT8 on these references; low-order INT8 can regress; validation-selected test failures are retained |
 | End-to-end artifact cost | [FastText matched controls](fasttext_native_evidence_20261031/README.md) | Whole bundle saves only 0.46%; queries are slower; structural order adds no benefit on this model |
 | Construction/runtime resources | [Implementation boundaries](native_ngram_memory.md) | Structural planning copies full rows/maps offline; cache budgets do not bound RSS; FastText retains an unused 200 MB hash allocation |
 

docs/quantized_structural_evidence_20261031/README.md

@@ -0,0 +1,118 @@
+# Quantization and original structural sharing on the same trained tables
+
+The original DATrie prefix/suffix planner and mark-compact retain an additional
+storage contribution after FP16 and per-row INT8 on these trained references.
+It is workload-dependent: INT8 compaction makes the 4-order GRU backend larger.
+Quantization and joint sharing are **approximate to original FP32 weights**;
+native packing preserves their encoded representative bytes and original IDs.
+
+## Matched storage results
+
+Three-seed **means**, fixed cosine and norm ratio 0.99 on decoded encoded rows.
+The denominator is the smallest same-encoding control among raw mmap, native
+identity, native lexical layout and exact encoded-byte dedup. Actual totals
+include the row table, forwarding map, codec metadata, INT8 scales and a common
+keys JSON. Positive values are extra space saved; negative values are regressions.
+
+| Reference / max order | Additional FP16 saving | Additional INT8 saving |
+| --- | ---: | ---: |
+| SGNS / 4 | +22.94% | +20.03% |
+| SGNS / 8 | +33.22% | +28.30% |
+| SGNS / 12 | +33.11% | +28.08% |
+| GRU / 4 | +5.26% | -4.56% |
+| GRU / 8 | +21.13% | +10.95% |
+| GRU / 12 | +25.42% | +15.10% |
+
+SGNS reports vector distortion, without a downstream task claim. The causal
+GRU has held-out character prediction metrics. The DATrie alternative replaces
+common keys JSON with an actual serialized exact-key index; its matched totals
+are also in [report.json](report.json) and [summary.json](summary.json). For INT8,
+those extra savings average 20.78/29.83/29.68% on SGNS and −4.90/12.14/16.90% on
+GRU at orders 4/8/12. Index lookup speed is outside this study.
+
+![Complete storage controls](storage-controls.svg)
+
+## Quality and selection boundaries
+
+The primary study selects the smallest backend within all validation budgets,
+separately by encoding: relative perplexity increase <= 0.5%, mean prediction
+KL <= 0.0001 nats, and top-1 changes <= 1% of positions. All choices are frozen
+before testing selected candidates; fixed raw FP16/INT8 test controls remain
+separate. Paired character-block intervals are reported without reselection.
+
+**20/27 primary validation choices pass all test budgets; seven fail mean KL.**
+The selected INT8 paths pass 8/9: all 4-order runs select raw INT8; 8-order seed 3
+fails KL; all three 12-order selections pass. The latter save an additional
+15.17–15.25% backend bytes over the best INT8 control, or 12.38–12.44% of the
+measured whole serving bundle. Their test perplexity increases 0.0051–0.0208%.
+These selected looser thresholds have maximum per-row relative L2 error of
+31–44%; small task changes do not establish small weight errors.
+
+The fixed default 0.99 bundles receive a **separate post-main-run test audit**
+in [fixed099-test-audit.json](fixed099-test-audit.json): **27/27 pass** the same
+budgets, with test perplexity changes -0.0050% to +0.0172% and maximum mean KL
+1.62e-05 nats. It does not replace any
+primary choices or failures and is not preregistered confirmatory evidence.
+Fixed thresholds were present in the original storage/validation sweep. Future
+quality confirmation needs independent data/checkpoints.
+
+The fixed 0.99 sweep reaches **14.31% maximum per-row relative L2 error** against
+original FP32, including INT8 rounding. “99% cosine” is not “1% vector error.”
+The query-weighted Frobenius curve below includes quantization's own error;
+additional error versus quantization-only is recorded separately. Relaxing the
+cosine guard on GRU adds little capacity benefit while increasing error tails.
+
+![Actual rate–distortion curves](rate-distortion.svg)
+
+Warm CPU gather p50 is **1.04–1.21× raw encoded mmap** for the fixed joint paths,
+including Python/NumPy dequantization. This study supports extra storage savings,
+with a query cost; it does not show a numeric gather speedup. First query means
+empty application cache on warmed filesystem pages, not disk-cold I/O.
+
+## Reproduce and inspect
+
+The primary run used clean source commit
+[`3a54de2`](https://github.com/initial-d/ngram-store/commit/3a54de2d8a7b97ee1432980c41b4edb0f64566d3),
+Python 3.11.15, PyTorch 2.11.0+cu130, NumPy 2.2.6, Linux x86-64 and an AMD EPYC
+7742 shared host. One PyTorch/interop/BLAS worker at nice 19, **CPU only**;
+wall time 8m48s. No weights were trained, external text downloaded or GPU
+programs touched. The corpus/checkpoint/row/source/library hashes and exact
+command are in the raw report. Build flags, tests and timing are retained in
+[build.txt](build.txt), [tests.txt](tests.txt), [run.sh](run.sh), [run.log](run.log)
+and [environment-extra.json](environment-extra.json). An initial launcher failed
+because `/usr/bin/time` was unavailable; [its error](resources-launch-failure.txt)
+is retained. Bash timing was used for the successful run.
+
+This is **18 cached public-text trained models and 486 storage variants**:
+SGNS/GRU orders 4/8/12, seeds 3/7/11; FP32/FP16/INT8, four encoded controls and
+five joint cosine thresholds (0.99/0.98/0.97/0.95/0.90), norm ratio fixed at 0.99.
+The production Linux DATrie planner is used; the portable reference planner is a
+separate smoke-test path. Every encoded row and original-ID forwarding result
+was checked. Planning uses rounded/dequantized rows before physical compaction.
+INT8 is symmetric nearest-even [-127,127], with its float32 scale inside the row.
+
+With the original cached inputs available, build and run the command from
+`run.sh` inside the pinned checkout. Summarize from the current repository:
+
+```bash
+python -m tools.analyze_quantized_structural \
+  --report docs/quantized_structural_evidence_20261031/report.json \
+  --output-dir artifacts/quantized-analysis --plots
+```
+
+A later finite-range INT8 guard prevents decode overflow near max float32.
+[codec-revision-check.json](codec-revision-check.json) verifies identical encoded
+and decoded bytes for all **54 trained encoding inputs**, using
+[verify_codec_revision.py](verify_codec_revision.py). Primary timings remain
+at the pinned study source. [audit_fixed099.py](audit_fixed099.py) reproduces the
+separate default-threshold audit in the pinned remote checkout.
+
+The measured whole GRU bundle sums backend/codec, common keys JSON and an actual
+serialized unchanged backbone/registry; the separate keys JSON duplicates key
+information held inside that registry, so it is a conservative evaluation
+bundle, not a minimal facade. Runtime/framework, training checkpoint, logs and
+builder/RSS costs are excluded. CPU exclusivity and frequency were not pinned.
+These small English SGNS/GRU references are not an official trained DeepSeek
+Engram checkpoint, a paper-scale reproduction, all-language evidence or a
+comparison with every learned/vector quantizer. Raw text, keys, weights and
+serving bundles remain local.

docs/quantized_structural_evidence_20261031/audit_fixed099.py

@@ -0,0 +1,82 @@
+"""Post-main-run fixed-default quality audit, separate from model selection.
+
+Uses the already declared 0.99 bundles; does not replace failed validation
+choices. This additional test audit is not preregistered confirmatory evidence.
+Run inside the pinned study checkout with the original locally cached inputs.
+"""
+from argparse import ArgumentParser, Namespace
+import hashlib
+import json
+from pathlib import Path
+import platform
+import shlex
+import subprocess
+import sys
+
+import torch
+
+from tools.benchmark_quantized_structural import score_bundle, file_sha
+from tools.benchmark_causal_lm_compression import text_splits, paired_ppl_interval
+from tools.causal_ngram_lm import CausalNgramLM, encode_segment, evaluate, metric_summary, prediction_delta
+
+
+def main():
+    p = ArgumentParser(description=__doc__)
+    p.add_argument("--study-root", type=Path, required=True)
+    p.add_argument("--output", type=Path, required=True)
+    args = p.parse_args()
+    torch.set_num_threads(1)
+    torch.set_num_interop_threads(1)
+    root = args.study_root
+    report_path = root / "results/report.json"
+    report = json.loads(report_path.read_text())
+    commit = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
+    if commit != report["commit"] or report["failures"] or len(report["runs"]) != 18:
+        raise ValueError("complete pinned primary study required")
+    corpus = Path("artifacts/native-ngram-20261018/corpus/tinyshakespeare.txt")
+    if file_sha(corpus) != report["corpus_sha256"]:
+        raise ValueError("corpus hash mismatch")
+    training = json.loads(Path("artifacts/causal-lm-compression-20261024/report.json").read_text())
+    text = text_splits(corpus.read_text(), training["settings"]["max_characters"])["test"]
+    cfg = Namespace(**report["settings"], native_library=Path("build/libmeft_rows.so"))
+    checks = []
+    for run in report["runs"]:
+        if run["category"] != "gru":
+            continue
+        checkpoint = Path("artifacts/causal-lm-compression-20261024") / f"order-{run['max_order']}-seed-{run['seed']}/reference.pt"
+        if file_sha(checkpoint) != run["checkpoint_sha256"]:
+            raise ValueError("checkpoint hash mismatch")
+        state = torch.load(checkpoint, map_location="cpu", weights_only=True)
+        model = CausalNgramLM(len(state["registry"]["characters"]) + 1, len(state["registry"]["grams"]), state["dim"])
+        model.load_state_dict(state["model_state"])
+        model.eval()
+        original = model.memory
+        segment = encode_segment(text, state["registry"])
+        baseline = evaluate(model, segment, cfg.burn_in, cfg.eval_chunk)
+        variants = {v["name"]: v for v in run["variants"]}
+        for encoding in ("fp32", "fp16", "int8"):
+            name = encoding + "-joint-0.99"
+            directory = root / "results" / f"gru-order-{run['max_order']}-seed-{run['seed']}" / name
+            if file_sha(directory / "table.men") != variants[name]["metadata"]["table_sha256"]:
+                raise ValueError("frozen bundle hash mismatch")
+            scores = score_bundle(model, original, directory, segment, cfg)
+            delta = prediction_delta(scores, baseline)
+            checks.append({"max_order": run["max_order"], "seed": run["seed"], "encoding": encoding,
+                "name": name, "checkpoint_sha256": run["checkpoint_sha256"],
+                "test": metric_summary(scores), "test_delta": delta,
+                "test_paired_block_interval": paired_ppl_interval(scores, baseline, seed=cfg.eval_seed),
+                "test_all_budgets_passed": delta["perplexity_relative_change"] <= cfg.max_ppl_increase
+                    and delta["mean_prediction_kl_nats"] <= cfg.max_kl and delta["top1_changed_fraction"] <= cfg.max_top1_flip})
+    result = {"category": "post-main-run fixed-default test audit; not preregistered confirmatory evidence",
+              "primary_report_sha256": file_sha(report_path), "commit": commit,
+              "script_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
+              "command": shlex.join([sys.executable, *sys.argv]), "python": sys.version,
+              "pytorch": torch.__version__, "platform": platform.platform(), "device": "cpu", "gpu": "unused",
+              "threads": 1, "interop_threads": 1, "primary_validation_choices_unchanged": True,
+              "quality_budgets": {"ppl_relative": cfg.max_ppl_increase, "mean_kl_nats": cfg.max_kl, "top1_flip_fraction": cfg.max_top1_flip},
+              "checks": checks}
+    args.output.write_text(json.dumps(result, indent=2) + "\n")
+
+
+if __name__ == "__main__":
+    main()

docs/quantized_structural_evidence_20261031/build.txt

@@ -0,0 +1,5 @@
+mkdir -p build
+cc -std=c11 -O2 -Wall -Wextra -Werror -fPIC -shared native/meft_rows.c -lz -o build/libmeft_rows.so
+cc -std=c11 -O2 -Wall -Wextra -Werror -Inative examples/native_gather.c native/meft_rows.c -lz -o build/native_gather
+mkdir -p build
+c++ -std=c++11 -O2 -Wall -Wextra -Werror -fPIC -shared -Isrc -Imylib/include native/meft_structural.cc -L./mylib/lib -ltrie -lpub -o build/libmeft_structural.so