From 329121f9ab7f6fee850179589e54a0d44fda6c2a Mon Sep 17 00:00:00 2001 From: initial-d Date: Sat, 31 Oct 2026 07:58:38 +0800 Subject: [PATCH] Connect n-gram embeddings and LLM memory through the learned table history --- README.md | 8 ++++++++ docs/artifact_card.md | 6 ++++++ docs/ngram_memory_research_plan.md | 19 +++++++++++++++++++ 3 files changed, 33 insertions(+) diff --git a/README.md b/README.md index 72ff8bb..99ea0df 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,14 @@ [![Paper](https://img.shields.io/badge/arXiv-2506.01254-b31b1b.svg)](https://arxiv.org/abs/2506.01254) [![Release](https://img.shields.io/github/v/release/initial-d/ngram-store?label=release)](https://github.com/initial-d/ngram-store/releases/latest) +**从早期 n-gram embedding，到大模型中的 learned n-gram 参数化记忆。** + +FastText uses learned subword vectors to build word representations; +Engram-style LLM memory retrieves learned token-n-gram features and fuses them +with hidden states. The application changes, while the need to index, store and +query trained vectors continues. NGramStore follows this thread through a +shared backend. Learned n-gram tables are one form of parametric memory. + Formerly **me_fasttext**. The engine grew from [Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management](https://arxiv.org/abs/2506.01254). The paper keeps its original title; later Engram integrations are software extensions. diff --git a/docs/artifact_card.md b/docs/artifact_card.md index da4af7a..802d68e 100644 --- a/docs/artifact_card.md +++ b/docs/artifact_card.md @@ -4,6 +4,12 @@ Formerly me_fasttext; the original Memory-Efficient FastText paper is the method origin, while native row storage and LLM integrations are later software work. +The historical thread runs from n-gram embeddings for word representations to +learned n-gram memory modules in LLMs. FastText and Engram illustrate these +application generations: both query trained vector tables, with different keys, +addressing and model computation. This table-based form of parametric memory is +the scope of the shared engine. + | Layer / component | What it provides | Boundary | | --- | --- | --- | | 1. Shared native C engine | Lossless compressed blocks, original-ID forwarding, mmap and gather | Frozen single-worker handles; caches do not bound RSS | diff --git a/docs/ngram_memory_research_plan.md b/docs/ngram_memory_research_plan.md index 5bb05a7..5e97692 100644 --- a/docs/ngram_memory_research_plan.md +++ b/docs/ngram_memory_research_plan.md @@ -1,5 +1,24 @@ # NGramStore engine roadmap and research plan +## Historical background: n-gram embeddings to LLM memory + +Early learned n-gram embeddings represent local patterns as trainable vectors. +FastText's subword vectors contribute to word representations. Newer Engram-style +LLM architectures retrieve learned token-n-gram features from tables and fuse +them with hidden states. This gives the same addressable-vector abstraction a +role as an explicit learned memory module alongside the LLM backbone. + +Here, n-gram parametric memory means learned lookup-table weights. This is one +branch of model memory, with a common storage/query problem across application +generations. The continuity is the table abstraction; character versus token +keys, hash versus exact addressing, training and context fusion remain distinct. + +NGramStore's historical development follows that abstraction: the original +me_fasttext method optimizes n-gram indexing and memory management for word +embeddings; the shared engine extends storage and queries to frozen learned +tables used by LLM memory modules. The later software evidence keeps its own +scope and does not rewrite the original paper's experiments. + ## Product contract and history NGramStore is a general-purpose backend for **learned n-gram embeddings**: