Skip to content

Integrate native structural storage with the legacy FastText model and CLI - #36

Merged
initial-d merged 6 commits into
mainfrom
codex/fasttext-native-fusion
Oct 30, 2026
Merged

initial-d merged 6 commits into
mainfrom
codex/fasttext-native-fusion

Conversation

@initial-d

@initial-d initial-d commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Dense me_fasttext .bin models can now be converted into immutable native serving bundles and queried through the original FastText facade. The optional make fasttext-native build accepts either .bin or a bundle directory with the existing word/sentence/classification CLI commands. Conversion uses the actual registered DATrie IDs, original structural planner and mark-compact forwarding; layout/exact modes preserve all input bytes and original float32 accumulation order.

The original build targets and dictionary save/load overloads remain available. Native input uses bounded gather chunks and a decoded cache, while dictionary/output state stays resident. Training and quantization use the retained dense model. This is Linux x86-64 source compatibility for this repository's unquantized models, not an old C++ binary ABI or official fastText/.ftz/.z converter.

Loaded dictionaries now skip new OOV rows beyond the trained matrix, direct subword helpers use registered DATrie IDs instead of hash buckets, HS prediction rebuilds its output tree, and zero-vector string sentence queries avoid NaN. Equivalence to historical valid queries is tested separately from these corrected behaviors.

Validation:

  • Actual legacy SGNS training on transformed existing enwik8: 90,380 × 64 float32 input rows, full reconstruction SHA equality, and identical held-out word/sentence bytes in nine fresh-process dense/layout/exact runs.
  • Historical main (bea4302) vs new dense vs native: identical outputs on 1,024 registered-word and 128 sentence queries using the same actual trained model.
  • Linux ASan/UBSan/LeakSanitizer facade fixtures: all input rows, UTF-8/OOV, softmax/HS scores, repeated/chunked float32 adds, immutable/failure/reload guards. Bundled trie archives remain uninstrumented.
  • Installed wheel checks validate the converter import/entry point without Torch. Final exact-head CI includes the new sanitized facade job.

Measured limits: structural native input saves 3.18%, but the whole bundle saves only 0.46%. Exact sharing finds zero identical rows; matched identity row order is 103,371 bytes smaller. Native hot word/sentence queries are slower than dense, and first-pass decoding is substantially slower. Anonymous RSS falls about 18.9 MiB while total RSS falls only 1.34 MiB after touched mmap pages. This is functional fusion evidence, not a universal compression/speed claim or an Engram task-quality result.

Usage · Raw trained results, matched layout controls, historical comparison and preserved initial failure. All remote work used CPU only in the maintainer's directory; no GPU process was touched.

@initial-d
initial-d marked this pull request as ready for review October 30, 2026 16:27
@initial-d
initial-d merged commit 67fa5fe into main Oct 30, 2026
18 checks passed
@initial-d
initial-d deleted the codex/fasttext-native-fusion branch October 30, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant