Repository navigation
Integrate native structural storage with the legacy FastText model and CLI - #36
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dense me_fasttext
.binmodels can now be converted into immutable native serving bundles and queried through the original FastText facade. The optionalmake fasttext-nativebuild accepts either.binor a bundle directory with the existing word/sentence/classification CLI commands. Conversion uses the actual registered DATrie IDs, original structural planner and mark-compact forwarding; layout/exact modes preserve all input bytes and original float32 accumulation order.The original build targets and dictionary save/load overloads remain available. Native input uses bounded gather chunks and a decoded cache, while dictionary/output state stays resident. Training and quantization use the retained dense model. This is Linux x86-64 source compatibility for this repository's unquantized models, not an old C++ binary ABI or official fastText/
.ftz/.zconverter.Loaded dictionaries now skip new OOV rows beyond the trained matrix, direct subword helpers use registered DATrie IDs instead of hash buckets, HS prediction rebuilds its output tree, and zero-vector string sentence queries avoid NaN. Equivalence to historical valid queries is tested separately from these corrected behaviors.
Validation:
bea4302) vs new dense vs native: identical outputs on 1,024 registered-word and 128 sentence queries using the same actual trained model.Measured limits: structural native input saves 3.18%, but the whole bundle saves only 0.46%. Exact sharing finds zero identical rows; matched identity row order is 103,371 bytes smaller. Native hot word/sentence queries are slower than dense, and first-pass decoding is substantially slower. Anonymous RSS falls about 18.9 MiB while total RSS falls only 1.34 MiB after touched mmap pages. This is functional fusion evidence, not a universal compression/speed claim or an Engram task-quality result.
Usage · Raw trained results, matched layout controls, historical comparison and preserved initial failure. All remote work used CPU only in the maintainer's directory; no GPU process was touched.