Repository navigation
Clarify the general n-gram embedding engine and training-to-serving roadmap - #38
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NGramStore's scope was split between a FastText origin, Engram adapters and an outdated high-order research plan. This change makes the shared frozen n-gram embedding table the engine contract, keeps DATrie/mark-compact optional, and presents FastText and Engram as applications with their original model semantics.
It explains the separate costs of bucket parameter sharing, full distinct-key tables and quantization/approximate sharing. Existing trained buckets remain shared; exact indexing requires distinct source rows and unique keys. The recommended lifecycle covers direct frozen-table export and stable-vocabulary training followed by frozen export, without implying native gradient support.
The roadmap orders sparse-query cost, independently measured structural gains, construction/process memory and external reuse, with trained same-source controls and historical compatibility gates. Current slow/no-benefit results remain visible, and obsolete pre-rename/guard-fix guidance is removed. Package description and guide navigation follow the same positioning.
Validation: all 69 local Markdown link targets in the five edited guides exist;
git diff --checkpasses;pyproject.tomlparses and preserves themeft-memorydistribution/version. No runtime changes, new benchmark claims or GPU runs. Existing CI checks integration and compatibility.