From db6155268a28af07b6f040ed88a119e24582a0a7 Mon Sep 17 00:00:00 2001 From: Don Hardman Date: Wed, 21 Oct 2026 10:21:31 +0700 Subject: [PATCH 1/5] docs(readme): document Manticore Columnar Library workflows README.md now provides comprehensive installation, build, usage, benchmark, release, and contribution documentation for Manticore Columnar Library. - Document columnar, indexing, vector search, and embedding capabilities - Replace installation and build guidance with Docker, Debian/Ubuntu, and source workflows - Expand platform requirements, compatibility, benchmarks, releases, and contribution guidance --- README.md | 178 +++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 135 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index dd2ba2a1..c04d2ae2 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@

- Manicore Search Logo + Manticore Search logo

@@ -8,82 +8,174 @@ Manticore Columnar Library -Manticore Columnar Library is a column-oriented storage and secondary indexing library, aiming to provide **decent performance with low memory footprint at big data volume**. -When used in combination with [Manticore Search](https://github.com/manticoresoftware/manticoresearch) can be beneficial for those looking for: -1. log analytics including rich free text search capabities (which is missing in e.g. [Clickhouse](https://github.com/ClickHouse/ClickHouse) - great tool for metrics analytics) -2. faster / low resource consumption log/metrics analytics. Since the library and Manticore Search are both written in C++ with low level optimizations in mind, in many cases the performance / RAM consumption is better than in Lucene / SOLR / Elasticsearch -3. running log / metric analytics in docker / kubernetes. Manticore Search + the library can work with as little as 30 megabytes of RAM which Elasticsearch / Clickhouse can't. It also starts in less than a second or a few seconds in the worst case. Since the overhead is so little you can afford having more nodes of Manticore Search + the library than Elasticsearch. More nodes and quicker start means higher high availability and agility. -4. powerful SQL for logs/metrics analytics and everything else [Manticore Search](https://github.com/manticoresoftware/manticoresearch) can give you +

+ Columnar storage, secondary indexes and KNN vector search for Manticore Search.
+ Query data that does not fit in RAM, filter it quickly, and search it by meaning. +

+ +

+ CI + Latest release + License: Apache-2.0 +

+ +Manticore Columnar Library (MCL) is a set of C++ libraries, plus a Rust embeddings library, that Manticore Search loads at startup. It provides the engine behind: + +| Feature | What it gives you | How it works | Manual | +|---|---|---|---| +| **Columnar storage** (`columnar/`) | Lower memory use on large tables: attributes stay on disk instead of in RAM | Each attribute is stored as its own column, in compressed blocks of 65,536 values. The encoding is chosen per block, and integers are packed with [FastPFOR](https://github.com/fast-pack/FastPFor) and [StreamVByte](https://github.com/fast-pack/streamvbyte) codecs | [Row-wise and columnar storage](https://manual.manticoresearch.com/Creating_a_table/Data_types#Row-wise-and-columnar-attribute-storages) | +| **Secondary indexes** (`secondary/`) | Faster filtering on attributes, enabled by default | Compact learned index built on the [PGM-index](https://github.com/gvinciguerra/PGM-index) | [Automatic secondary indexes](https://manual.manticoresearch.com/Introduction#Automatic-secondary-indexes) | +| **KNN vector search** (`knn/`) | Similarity, semantic and hybrid search over `float_vector` attributes | HNSW graphs built on [hnswlib](https://github.com/nmslib/hnswlib), with L2, inner-product and cosine distance, plus optional 8-bit and 1-bit quantization | [KNN vector search](https://manual.manticoresearch.com/Searching/KNN) | +| **Auto embeddings** (`embeddings/`) | Text is turned into vectors inside Manticore, with no separate model service | Local models run on ONNX Runtime or [Candle](https://github.com/huggingface/candle). Remote models use the OpenAI, Voyage and Jina APIs | [Auto embeddings](https://manual.manticoresearch.com/Searching/KNN#Auto-Embeddings-%28Recommended%29) | + +> [!NOTE] +> MCL is not a general-purpose standalone library. Its C++ interfaces (`columnar/columnar.h`, `secondary/secondary.h`, `knn/knn.h`) exist for Manticore Search. Each one carries a `LIB_VERSION` that must exactly match the version the daemon was built against. To use these features, install Manticore Search. + +## How it fits into Manticore Search + +```mermaid +flowchart LR + client["Clients
SQL ยท HTTP JSON"] --> searchd["Manticore Search daemon
(searchd)"] + searchd -- dlopen --> col["lib_manticore_columnar
columnar attribute storage"] + searchd -- dlopen --> si["lib_manticore_secondary
secondary indexes"] + searchd -- dlopen --> knn["lib_manticore_knn
HNSW vector search"] + knn -- loads --> emb["lib_manticore_knn_embeddings
auto embeddings (Rust)"] +``` + +At startup, `searchd` looks for these modules in its modules directory, for example `/usr/share/manticore/modules` on Linux. `indexer` and `indextool` load the same modules. On x86_64 it prefers the `_avx512` build, then the `_avx2` build, then the baseline build, depending on what the CPU supports. If a module is missing, the daemon still runs, but that feature is unavailable. ## Getting started -### Requirements +MCL ships inside Manticore Search. Since Manticore 25.0.0, the `manticore` package for Debian, Ubuntu and RHEL-based systems is a bundle that already includes `manticore-columnar-lib`. The `manticore` Homebrew formula from `manticoresoftware/tap` and the official Docker image include the MCL modules too. -1. CPU: with SSE >= 4.2 -2. Architecture: arm64 or x86_64 -3. OS: Debian-based OS (Debian, Ubuntu, Mint), RHEL-based (RHEL, Centos, Alma, Oracle Linux, Amazon Linux) OS, Windows, MacOS +**Try it in Docker:** + +```bash +docker run --name manticore --rm -d manticoresearch/manticore && until docker logs manticore 2>&1 | grep -q "accepting connections"; do sleep 1; done && docker exec -it manticore mysql && docker stop manticore +``` + +This sandbox keeps no data. When you exit the MySQL client, the container stops and is removed. + +**Install on Ubuntu or Debian** ([other platforms](https://manual.manticoresearch.com/Installation/Installation)): -### Installation from yum/apt repositories -#### Ubuntu, Debian: ```bash wget https://repo.manticoresearch.com/manticore-repo.noarch.deb -sudo dpkg -i manticore-repo.noarch.deb +sudo apt install ./manticore-repo.noarch.deb sudo apt update -sudo apt install manticore-columnar-lib +sudo apt install manticore ``` -#### Centos: -```bash -sudo yum install https://repo.manticoresearch.com/manticore-repo.noarch.rpm -sudo yum install manticore-columnar-lib +Check that the libraries were loaded: `searchd -v` lists the columnar, secondary, knn and embeddings versions. Output from the `manticoresearch/manticore:29.9.0` image: + ``` +Manticore 29.9.0 58e88d5ab@26091108 (columnar 14.2.1 cb282a2@26090615) (secondary 14.2.1 cb282a2@26090615) (knn 14.2.1 cb282a2@26090615) (embeddings 1.1.1 cb282a2@26090615) +``` + +Then create a table that stores its attributes in columnar storage and has a vector column: + +```sql +CREATE TABLE products(title TEXT, price FLOAT, + emb FLOAT_VECTOR KNN_TYPE='hnsw' KNN_DIMS='4' HNSW_SIMILARITY='l2') ENGINE='columnar'; + +INSERT INTO products(id, title, price, emb) VALUES + (1, 'red shoes', 49.9, (0.1, 0.2, 0.3, 0.4)), + (2, 'blue shirt', 19.5, (0.9, 0.1, 0.1, 0.1)); + +SELECT id, title, price FROM products WHERE price > 20; +SELECT id, knn_dist() FROM products WHERE knn(emb, 2, (0.1, 0.2, 0.3, 0.4)); +``` + +You can also make only some attributes columnar with `engine='columnar'` on those attributes. Plain tables use the `columnar_attrs` setting instead. See [how to switch between the storages](https://manual.manticoresearch.com/Creating_a_table/Data_types#How-to-switch-between-the-storages). + +## Requirements and limitations + +- **Platforms.** MCL runs wherever Manticore Search packages are available: Debian-based and RHEL-based Linux and macOS on x86_64 and arm64, and Windows on x86_64. See the Manticore [installation guide](https://manual.manticoresearch.com/Installation/Installation) for supported distribution releases. +- **CPU.** On Linux and macOS, the baseline x86_64 build targets `x86-64-v2`; MSVC and clang-cl baseline builds use `/arch:AVX`. Columnar storage and secondary indexes need SSE 4.2; otherwise the daemon reports `MCL requires a CPU that supports SSE 4.2`. KNN search also needs POPCNT and reports `MCL requires a CPU that supports SSE 4.2 and POPCNT instruction`. In both cases `searchd` logs a warning and keeps running without that library. AVX2 and AVX-512 builds are used automatically when the CPU supports them. On arm64, the SIMD code runs through [SIMDe](https://github.com/simd-everywhere/simde). +- **Version matching.** Each library exposes an API number, `LIB_VERSION`. If it differs from the one the daemon was built with, the daemon refuses to load the library, for example `daemon requires columnar library v29 (trying to load v28)`. Upgrade or downgrade Manticore and MCL together. The `manticore` bundle package does this for you. +- **Columnar is a trade-off.** If all your attributes fit in RAM, the default row-wise storage usually gives lower latency. Columnar storage is designed for data that does not fit in memory. +- **KNN memory.** HNSW indexes must be fully loaded into memory. Quantization reduces this footprint. Float vectors and KNN search are only supported in real-time tables. +- **Windows builds.** Windows DLLs are cross-compiled from Linux with clang. A native MSVC build of this repository is not supported, because MSVC rejects designated initializers under C++17. + +## Building from source + +Most users do not need this: the libraries come with Manticore Search. Build from source to work on MCL itself. + +The C++ libraries need CMake 3.21 or newer, a C++17 compiler (GCC or Clang) and network access. During configuration, CMake downloads Manticore's forks of FastPFOR, StreamVByte, PGM-index and hnswlib from GitHub, unless you provide them in a local bundle directory (`LIBS_BUNDLE`). These are the build commands CI runs on Linux x86_64, Linux aarch64 and macOS arm64. The x86_64 job also sets `-DBUILD_TESTING=ON`, and a separate job cross-compiles the Windows x64 build from Linux: -`searchd -v` should include `columnar x.y.z`, e.g.: ```bash -root@srv# searchd -v -Manticore 6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130) +git clone https://github.com/manticoresoftware/columnar.git +cd columnar +cmake -B build -DBUILD_TESTING=OFF +cmake --build build --config RelWithDebInfo --target columnar_lib secondary_index knn_lib --parallel ``` -### Basic usage: -1. Read https://manual.manticoresearch.com/Creating_a_table/Data_types#Row-wise-and-columnar-attribute-storages -2. Create plain or real-time table specifying that the columnar storage should be used +This produces `lib_manticore_columnar`, `lib_manticore_secondary` and `lib_manticore_knn`. Building the explicit targets skips the extra AVX2 and AVX-512 builds. A plain `cmake --build build` on x86_64 includes them. + +
+Embeddings library, tests, and running a local build with Manticore + +- **Embeddings.** The embeddings library is a Rust crate in [`embeddings/`](embeddings/). Add `-DBUILD_EMBEDDINGS_LOCALLY=ON` to build it from CMake; this requires Rust and Cargo, and CMake downloads ONNX Runtime. The `embeddings` job in [`.github/workflows/ci.yml`](.github/workflows/ci.yml) shows the full toolchain and the `cargo fmt`, `clippy` and `test` commands. +- **Tests.** With `-DBUILD_TESTING=ON`, the `knn_graphrepair_test`, `knn_multivector_slot_test` and `knn_embedding_span_test` targets build focused KNN tests that run with `ctest -R '^knn_'`. The full functional test suite runs from the Manticore Search repository against a daemon. +- **Running with Manticore.** `searchd` takes module paths from the `LIB_MANTICORE_COLUMNAR`, `LIB_MANTICORE_SECONDARY`, `LIB_MANTICORE_KNN` and `LIB_MANTICORE_KNN_EMBEDDINGS` environment variables. Point them at your build outputs. The Manticore Search repository also includes MCL as the `mcl` git submodule. Configure it with `-DMCL_LOCAL_RUNTIME=ON` to build the daemon and MCL together. See [Compiling from sources](https://manual.manticoresearch.com/Installation/Compiling_from_sources). + +
## Benchmarks -### Log analytics - 6x faster than Elasticsearch +In 2022, the Manticore team published these comparisons of Manticore Search with columnar storage against Elasticsearch and ClickHouse on [db-benchmarks.com](https://db-benchmarks.com/). The test framework is [open source](https://github.com/db-benchmarks/db-benchmarks). The figures below are the geometric mean of per-query response-time ratios from the published charts. They reflect the software versions of that time. + +| Dataset | vs Elasticsearch | vs ClickHouse | +|---|---|---| +| [10M Nginx logs](https://db-benchmarks.com/test-logs10m/) | [6.52x faster](https://db-benchmarks.com/test-logs10m/#elasticsearch-tuned-vs-manticore-search-columnar-storage) (tuned) | [1.40x faster](https://db-benchmarks.com/test-logs10m/#clickhouse-vs-manticore-search-columnar-storage) | +| [110M Hacker News comments](https://db-benchmarks.com/test-hn/) | [5.06x faster](https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-elasticsearch) (tuned) | [11.60x faster](https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-clickhouse) | +| [1.7B NYC taxi rides](https://db-benchmarks.com/test-taxi/) | [3.97x faster](https://db-benchmarks.com/test-taxi/#manticore-search-vs-elasticsearch) (tuned) | [1.81x faster](https://db-benchmarks.com/test-taxi/#manticore-search-vs-clickhouse) | + +Most comparisons include queries where Manticore is slower; the per-query tables show them. In the 10M logs test, Elasticsearch timed out on 2 queries, which were excluded from its score. + +
+Charts + +**10M Nginx logs, compared with Elasticsearch (tuned)** -https://db-benchmarks.com/test-logs10m/#elasticsearch-tuned-vs-manticore-search-columnar-storage +![Manticore Search columnar vs Elasticsearch on 10M Nginx logs](https://db-benchmarks.com/test-logs10m/est_msc.png) -![logs_es_msc](https://db-benchmarks.com/test-logs10m/est_msc.png) +**10M Nginx logs, compared with ClickHouse** -### Log analytics - 1.4x faster than Clickhouse +![Manticore Search columnar vs ClickHouse on 10M Nginx logs](https://db-benchmarks.com/test-logs10m/ch_msc.png) -https://db-benchmarks.com/test-logs10m/#clickhouse-vs-manticore-search-columnar-storage +**110M Hacker News comments, compared with Elasticsearch (tuned)** -![logs_es_ch](https://db-benchmarks.com/test-logs10m/ch_msc.png) +![Manticore Search columnar vs Elasticsearch on 110M Hacker News comments](https://db-benchmarks.com/test-hn/msc_es.png) -### Medium data - 110M Hackernews comments - 5x faster than Elasticsearch +**110M Hacker News comments, compared with ClickHouse** -https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-elasticsearch +![Manticore Search columnar vs ClickHouse on 110M Hacker News comments](https://db-benchmarks.com/test-hn/msc_ch.png) -![hn_es_msc](https://db-benchmarks.com/test-hn/msc_es.png) +**1.7B NYC taxi rides, compared with Elasticsearch (tuned)** -### Medium data - 110M Hackernews comments - 11x faster than Clickhouse +![Manticore Search columnar vs Elasticsearch on 1.7B NYC taxi rides](https://db-benchmarks.com/test-taxi/ms_es.png) -https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-clickhouse +**1.7B NYC taxi rides, compared with ClickHouse** -![hn_msc_ch](https://db-benchmarks.com/test-hn/msc_ch.png) +![Manticore Search columnar vs ClickHouse on 1.7B NYC taxi rides](https://db-benchmarks.com/test-taxi/ms_ch.png) -### Big data - 1.7B NYC taxi rides - 4x faster than Elasticsearch +
-https://db-benchmarks.com/test-taxi/#manticore-search-vs-elasticsearch +Further reading on the Manticore blog: [About columnar storage](https://manticoresearch.com/blog/mcl/), [mmap for columnar KNN rescoring](https://manticoresearch.com/blog/columnar-mmap/), [vector quantization](https://manticoresearch.com/blog/quantization/), [faster HNSW search](https://manticoresearch.com/blog/knn-hnsw-performance/) and [ONNX embeddings](https://manticoresearch.com/blog/onnx-embeddings-speedup/). -![taxi_ms_es](https://db-benchmarks.com/test-taxi/ms_es.png) +## Versions and releases -### Big data - 1.7B NYC taxi rides - 1.8x faster than Clickhouse +- The latest GitHub release is [14.2.1](https://github.com/manticoresoftware/columnar/releases/tag/14.2.1). It is the version in the current `manticore-columnar-lib` packages and in Manticore Search 29.9.0. +- Pushes to `master` that change code get a version tag automatically from [`.github/workflows/auto_version.yml`](.github/workflows/auto_version.yml), so `master` can be ahead of the latest release. The `project()` version in `CMakeLists.txt` is a placeholder that the packaging workflow replaces. +- Binary packages are built and published by the Manticore Search repository, not attached to GitHub releases here. +- For release notes, see [GitHub Releases](https://github.com/manticoresoftware/columnar/releases) and the [Manticore Search changelog](https://manual.manticoresearch.com/Changelog), where MCL updates are listed. [`Changelog.md`](Changelog.md) in this repository covers only older versions. -https://db-benchmarks.com/test-taxi/#manticore-search-vs-clickhouse +## Contributing, support and security -![taxi_ms_ch](https://db-benchmarks.com/test-taxi/ms_ch.png) +- **Bugs and feature requests:** [open an issue](https://github.com/manticoresoftware/columnar/issues/new/choose). Issues about Manticore Search behavior in general belong in the [Manticore Search repository](https://github.com/manticoresoftware/manticoresearch/issues). +- **Contributing:** this repository has no separate guide. The [Manticore Search contributing guide](https://github.com/manticoresoftware/manticoresearch/blob/main/CONTRIBUTING.md) applies. Pull requests run the checks in [`.github/workflows/ci.yml`](.github/workflows/ci.yml). +- **Security:** do not report vulnerabilities in public issues. Report them privately through [Manticore Search's private vulnerability reporting](https://github.com/manticoresoftware/manticoresearch/security/advisories/new). +## License +Manticore Columnar Library is licensed under the [Apache License 2.0](LICENSE). Binary packages also contain third-party components; see [`LICENSE-binary`](LICENSE-binary) and [`NOTICE-binary`](NOTICE-binary). [`licenses-binary/`](licenses-binary/) holds the licenses of the bundled C++ components (FastPFOR, PGM-index, hnswlib, SIMDe and StreamVByte). The Rust dependencies of the embeddings library are listed in [`embeddings/Cargo.lock`](embeddings/Cargo.lock). From 0a874252d8f6de25ca1b8d5be472e029634b4276 Mon Sep 17 00:00:00 2001 From: Don Hardman Date: Wed, 21 Oct 2026 10:56:57 +0700 Subject: [PATCH 2/5] docs(readme): clarify storage limits and benchmark metrics - Document JSON secondary-index configuration and columnar attribute limitations - Clarify benchmark summary methodology --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c04d2ae2..24d11a40 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ Manticore Columnar Library (MCL) is a set of C++ libraries, plus a Rust embeddin | Feature | What it gives you | How it works | Manual | |---|---|---|---| | **Columnar storage** (`columnar/`) | Lower memory use on large tables: attributes stay on disk instead of in RAM | Each attribute is stored as its own column, in compressed blocks of 65,536 values. The encoding is chosen per block, and integers are packed with [FastPFOR](https://github.com/fast-pack/FastPFor) and [StreamVByte](https://github.com/fast-pack/streamvbyte) codecs | [Row-wise and columnar storage](https://manual.manticoresearch.com/Creating_a_table/Data_types#Row-wise-and-columnar-attribute-storages) | -| **Secondary indexes** (`secondary/`) | Faster filtering on attributes, enabled by default | Compact learned index built on the [PGM-index](https://github.com/gvinciguerra/PGM-index) | [Automatic secondary indexes](https://manual.manticoresearch.com/Introduction#Automatic-secondary-indexes) | +| **Secondary indexes** (`secondary/`) | Faster filtering on attributes. Secondary indexes are on by default for numeric and string attributes; JSON attributes can be enabled with `json_secondary_indexes` | Compact learned index built on the [PGM-index](https://github.com/gvinciguerra/PGM-index) | [Automatic secondary indexes](https://manual.manticoresearch.com/Introduction#Automatic-secondary-indexes) | | **KNN vector search** (`knn/`) | Similarity, semantic and hybrid search over `float_vector` attributes | HNSW graphs built on [hnswlib](https://github.com/nmslib/hnswlib), with L2, inner-product and cosine distance, plus optional 8-bit and 1-bit quantization | [KNN vector search](https://manual.manticoresearch.com/Searching/KNN) | | **Auto embeddings** (`embeddings/`) | Text is turned into vectors inside Manticore, with no separate model service | Local models run on ONNX Runtime or [Candle](https://github.com/huggingface/candle). Remote models use the OpenAI, Voyage and Jina APIs | [Auto embeddings](https://manual.manticoresearch.com/Searching/KNN#Auto-Embeddings-%28Recommended%29) | @@ -93,6 +93,7 @@ You can also make only some attributes columnar with `engine='columnar'` on thos - **CPU.** On Linux and macOS, the baseline x86_64 build targets `x86-64-v2`; MSVC and clang-cl baseline builds use `/arch:AVX`. Columnar storage and secondary indexes need SSE 4.2; otherwise the daemon reports `MCL requires a CPU that supports SSE 4.2`. KNN search also needs POPCNT and reports `MCL requires a CPU that supports SSE 4.2 and POPCNT instruction`. In both cases `searchd` logs a warning and keeps running without that library. AVX2 and AVX-512 builds are used automatically when the CPU supports them. On arm64, the SIMD code runs through [SIMDe](https://github.com/simd-everywhere/simde). - **Version matching.** Each library exposes an API number, `LIB_VERSION`. If it differs from the one the daemon was built with, the daemon refuses to load the library, for example `daemon requires columnar library v29 (trying to load v28)`. Upgrade or downgrade Manticore and MCL together. The `manticore` bundle package does this for you. - **Columnar is a trade-off.** If all your attributes fit in RAM, the default row-wise storage usually gives lower latency. Columnar storage is designed for data that does not fit in memory. +- **Columnar attribute limits.** JSON attributes cannot use columnar storage; keep them row-wise in the same table. Columnar attributes cannot be changed with `UPDATE`; use `REPLACE`. - **KNN memory.** HNSW indexes must be fully loaded into memory. Quantization reduces this footprint. Float vectors and KNN search are only supported in real-time tables. - **Windows builds.** Windows DLLs are cross-compiled from Linux with clang. A native MSVC build of this repository is not supported, because MSVC rejects designated initializers under C++17. @@ -122,7 +123,7 @@ This produces `lib_manticore_columnar`, `lib_manticore_secondary` and `lib_manti ## Benchmarks -In 2022, the Manticore team published these comparisons of Manticore Search with columnar storage against Elasticsearch and ClickHouse on [db-benchmarks.com](https://db-benchmarks.com/). The test framework is [open source](https://github.com/db-benchmarks/db-benchmarks). The figures below are the geometric mean of per-query response-time ratios from the published charts. They reflect the software versions of that time. +In 2022, the Manticore team published these comparisons of Manticore Search with columnar storage against Elasticsearch and ClickHouse on [db-benchmarks.com](https://db-benchmarks.com/). The test framework is [open source](https://github.com/db-benchmarks/db-benchmarks). The figures below are the "Relative query processing time" summaries from the published charts, which are based on geometric means of per-query response-time ratios. They reflect the software versions of that time. | Dataset | vs Elasticsearch | vs ClickHouse | |---|---|---| From f1046c29c0e61b69d4bf047afad2f8994a5a3d44 Mon Sep 17 00:00:00 2001 From: Don Hardman Date: Thu, 22 Oct 2026 10:34:31 +0700 Subject: [PATCH 3/5] docs(readme): clarify MCL positioning and build guidance The README now presents MCL as a standalone storage, indexing, and vector search library, and clarifies its relationship with Manticore Search, platform compatibility, embeddings, and source builds. --- README.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 24d11a40..e9139c8e 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@

- Columnar storage, secondary indexes and KNN vector search for Manticore Search.
+ Column-oriented storage, secondary indexes and KNN vector search in one library.
Query data that does not fit in RAM, filter it quickly, and search it by meaning.

@@ -19,19 +19,23 @@ License: Apache-2.0

-Manticore Columnar Library (MCL) is a set of C++ libraries, plus a Rust embeddings library, that Manticore Search loads at startup. It provides the engine behind: +Manticore Columnar Library (MCL) is a column-oriented storage, secondary indexing and vector search library, aiming to provide decent performance with a low memory footprint at big data volumes. It consists of C++ libraries plus a Rust embeddings library: | Feature | What it gives you | How it works | Manual | |---|---|---|---| | **Columnar storage** (`columnar/`) | Lower memory use on large tables: attributes stay on disk instead of in RAM | Each attribute is stored as its own column, in compressed blocks of 65,536 values. The encoding is chosen per block, and integers are packed with [FastPFOR](https://github.com/fast-pack/FastPFor) and [StreamVByte](https://github.com/fast-pack/streamvbyte) codecs | [Row-wise and columnar storage](https://manual.manticoresearch.com/Creating_a_table/Data_types#Row-wise-and-columnar-attribute-storages) | | **Secondary indexes** (`secondary/`) | Faster filtering on attributes. Secondary indexes are on by default for numeric and string attributes; JSON attributes can be enabled with `json_secondary_indexes` | Compact learned index built on the [PGM-index](https://github.com/gvinciguerra/PGM-index) | [Automatic secondary indexes](https://manual.manticoresearch.com/Introduction#Automatic-secondary-indexes) | | **KNN vector search** (`knn/`) | Similarity, semantic and hybrid search over `float_vector` attributes | HNSW graphs built on [hnswlib](https://github.com/nmslib/hnswlib), with L2, inner-product and cosine distance, plus optional 8-bit and 1-bit quantization | [KNN vector search](https://manual.manticoresearch.com/Searching/KNN) | -| **Auto embeddings** (`embeddings/`) | Text is turned into vectors inside Manticore, with no separate model service | Local models run on ONNX Runtime or [Candle](https://github.com/huggingface/candle). Remote models use the OpenAI, Voyage and Jina APIs | [Auto embeddings](https://manual.manticoresearch.com/Searching/KNN#Auto-Embeddings-%28Recommended%29) | +| **Auto embeddings** (`embeddings/`) | Text is turned into vectors in-process, with no separate model service | Local models run on ONNX Runtime or [Candle](https://github.com/huggingface/candle). Remote models use the OpenAI, Voyage and Jina APIs | [Auto embeddings](https://manual.manticoresearch.com/Searching/KNN#Auto-Embeddings-%28Recommended%29) | -> [!NOTE] -> MCL is not a general-purpose standalone library. Its C++ interfaces (`columnar/columnar.h`, `secondary/secondary.h`, `knn/knn.h`) exist for Manticore Search. Each one carries a `LIB_VERSION` that must exactly match the version the daemon was built against. To use these features, install Manticore Search. +When used in combination with [Manticore Search](https://github.com/manticoresoftware/manticoresearch), MCL can be beneficial for those looking for: -## How it fits into Manticore Search +1. log analytics with rich full-text search, which metrics-focused tools such as [ClickHouse](https://github.com/ClickHouse/ClickHouse) lack; +2. faster log and metrics analytics with lower resource use. MCL and Manticore Search are both written in C++ with low-level optimizations in mind; see the [benchmarks](#benchmarks) against Elasticsearch and ClickHouse; +3. semantic and hybrid search, with embeddings generated in-process; +4. powerful SQL and everything else Manticore Search offers. + +## Using MCL with Manticore Search ```mermaid flowchart LR @@ -46,7 +50,7 @@ At startup, `searchd` looks for these modules in its modules directory, for exam ## Getting started -MCL ships inside Manticore Search. Since Manticore 25.0.0, the `manticore` package for Debian, Ubuntu and RHEL-based systems is a bundle that already includes `manticore-columnar-lib`. The `manticore` Homebrew formula from `manticoresoftware/tap` and the official Docker image include the MCL modules too. +The easiest way to use MCL is with Manticore Search. Since Manticore 25.0.0, the `manticore` package for Debian, Ubuntu and RHEL-based systems is a bundle that already includes `manticore-columnar-lib`. The `manticore` Homebrew formula from `manticoresoftware/tap` and the official Docker image include the MCL modules too. **Try it in Docker:** @@ -89,9 +93,9 @@ You can also make only some attributes columnar with `engine='columnar'` on thos ## Requirements and limitations -- **Platforms.** MCL runs wherever Manticore Search packages are available: Debian-based and RHEL-based Linux and macOS on x86_64 and arm64, and Windows on x86_64. See the Manticore [installation guide](https://manual.manticoresearch.com/Installation/Installation) for supported distribution releases. +- **Platforms.** Debian-based and RHEL-based Linux and macOS on x86_64 and arm64, and Windows on x86_64. For supported distribution releases, see the Manticore [installation guide](https://manual.manticoresearch.com/Installation/Installation). - **CPU.** On Linux and macOS, the baseline x86_64 build targets `x86-64-v2`; MSVC and clang-cl baseline builds use `/arch:AVX`. Columnar storage and secondary indexes need SSE 4.2; otherwise the daemon reports `MCL requires a CPU that supports SSE 4.2`. KNN search also needs POPCNT and reports `MCL requires a CPU that supports SSE 4.2 and POPCNT instruction`. In both cases `searchd` logs a warning and keeps running without that library. AVX2 and AVX-512 builds are used automatically when the CPU supports them. On arm64, the SIMD code runs through [SIMDe](https://github.com/simd-everywhere/simde). -- **Version matching.** Each library exposes an API number, `LIB_VERSION`. If it differs from the one the daemon was built with, the daemon refuses to load the library, for example `daemon requires columnar library v29 (trying to load v28)`. Upgrade or downgrade Manticore and MCL together. The `manticore` bundle package does this for you. +- **Version matching.** Each library exposes an API number, `LIB_VERSION`. If it differs from the one the host application was built with, the library is not loaded. For example, `searchd` reports `daemon requires columnar library v29 (trying to load v28)`. With Manticore Search, upgrade or downgrade Manticore and MCL together. The `manticore` bundle package does this for you. - **Columnar is a trade-off.** If all your attributes fit in RAM, the default row-wise storage usually gives lower latency. Columnar storage is designed for data that does not fit in memory. - **Columnar attribute limits.** JSON attributes cannot use columnar storage; keep them row-wise in the same table. Columnar attributes cannot be changed with `UPDATE`; use `REPLACE`. - **KNN memory.** HNSW indexes must be fully loaded into memory. Quantization reduces this footprint. Float vectors and KNN search are only supported in real-time tables. @@ -99,7 +103,7 @@ You can also make only some attributes columnar with `engine='columnar'` on thos ## Building from source -Most users do not need this: the libraries come with Manticore Search. Build from source to work on MCL itself. +Build from source to work on MCL or to integrate it into your own application. Packaged builds come with Manticore Search. The C++ libraries need CMake 3.21 or newer, a C++17 compiler (GCC or Clang) and network access. During configuration, CMake downloads Manticore's forks of FastPFOR, StreamVByte, PGM-index and hnswlib from GitHub, unless you provide them in a local bundle directory (`LIBS_BUNDLE`). These are the build commands CI runs on Linux x86_64, Linux aarch64 and macOS arm64. The x86_64 job also sets `-DBUILD_TESTING=ON`, and a separate job cross-compiles the Windows x64 build from Linux: From 8f95d3dcfa2054c6e30746a37fbd8273a3de5e07 Mon Sep 17 00:00:00 2001 From: Don Hardman Date: Thu, 22 Oct 2026 10:39:45 +0700 Subject: [PATCH 4/5] docs(readme): clarify library capabilities - Clarify library capabilities and embedding execution options - Simplify and revise the listed benefits of using MCL with Manticore Search --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index e9139c8e..96edb4e7 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@

- Column-oriented storage, secondary indexes and KNN vector search in one library.
+ Columnar storage, secondary indexes and KNN vector search libraries.
Query data that does not fit in RAM, filter it quickly, and search it by meaning.

@@ -26,13 +26,13 @@ Manticore Columnar Library (MCL) is a column-oriented storage, secondary indexin | **Columnar storage** (`columnar/`) | Lower memory use on large tables: attributes stay on disk instead of in RAM | Each attribute is stored as its own column, in compressed blocks of 65,536 values. The encoding is chosen per block, and integers are packed with [FastPFOR](https://github.com/fast-pack/FastPFor) and [StreamVByte](https://github.com/fast-pack/streamvbyte) codecs | [Row-wise and columnar storage](https://manual.manticoresearch.com/Creating_a_table/Data_types#Row-wise-and-columnar-attribute-storages) | | **Secondary indexes** (`secondary/`) | Faster filtering on attributes. Secondary indexes are on by default for numeric and string attributes; JSON attributes can be enabled with `json_secondary_indexes` | Compact learned index built on the [PGM-index](https://github.com/gvinciguerra/PGM-index) | [Automatic secondary indexes](https://manual.manticoresearch.com/Introduction#Automatic-secondary-indexes) | | **KNN vector search** (`knn/`) | Similarity, semantic and hybrid search over `float_vector` attributes | HNSW graphs built on [hnswlib](https://github.com/nmslib/hnswlib), with L2, inner-product and cosine distance, plus optional 8-bit and 1-bit quantization | [KNN vector search](https://manual.manticoresearch.com/Searching/KNN) | -| **Auto embeddings** (`embeddings/`) | Text is turned into vectors in-process, with no separate model service | Local models run on ONNX Runtime or [Candle](https://github.com/huggingface/candle). Remote models use the OpenAI, Voyage and Jina APIs | [Auto embeddings](https://manual.manticoresearch.com/Searching/KNN#Auto-Embeddings-%28Recommended%29) | +| **Auto embeddings** (`embeddings/`) | Text is turned into vectors by the library itself. Local models need no separate model service | Local models run on ONNX Runtime or [Candle](https://github.com/huggingface/candle). Remote models use the OpenAI, Voyage and Jina APIs | [Auto embeddings](https://manual.manticoresearch.com/Searching/KNN#Auto-Embeddings-%28Recommended%29) | When used in combination with [Manticore Search](https://github.com/manticoresoftware/manticoresearch), MCL can be beneficial for those looking for: -1. log analytics with rich full-text search, which metrics-focused tools such as [ClickHouse](https://github.com/ClickHouse/ClickHouse) lack; -2. faster log and metrics analytics with lower resource use. MCL and Manticore Search are both written in C++ with low-level optimizations in mind; see the [benchmarks](#benchmarks) against Elasticsearch and ClickHouse; -3. semantic and hybrid search, with embeddings generated in-process; +1. log analytics combined with rich full-text search; +2. faster analytics with lower memory use on large tables. The MCL C++ libraries and Manticore Search are both written with low-level optimizations, such as SIMD, in mind; see the [benchmarks](#benchmarks) against Elasticsearch and ClickHouse; +3. semantic and hybrid search, with embeddings generated from local models or the OpenAI, Voyage and Jina APIs; 4. powerful SQL and everything else Manticore Search offers. ## Using MCL with Manticore Search From 520817800915c8909f1e8689ded13b6f1b763372 Mon Sep 17 00:00:00 2001 From: Don Hardman Date: Thu, 22 Oct 2026 16:08:31 +0700 Subject: [PATCH 5/5] docs(readme): update installation and benchmark documentation The README now documents current package installation and benchmark comparisons, while identifying older charts and release information more accurately. - Update installation instructions to use the library package and clarify package replacement - Refresh benchmark values, links, versions, and historical chart labeling - Point version guidance to the latest GitHub release and matching shipped versions --- README.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 96edb4e7..e3009594 100644 --- a/README.md +++ b/README.md @@ -60,15 +60,17 @@ docker run --name manticore --rm -d manticoresearch/manticore && until docker lo This sandbox keeps no data. When you exit the MySQL client, the container stops and is removed. -**Install on Ubuntu or Debian** ([other platforms](https://manual.manticoresearch.com/Installation/Installation)): +**Install the library package on Ubuntu or Debian:** ```bash wget https://repo.manticoresearch.com/manticore-repo.noarch.deb sudo apt install ./manticore-repo.noarch.deb sudo apt update -sudo apt install manticore +sudo apt install manticore-columnar-lib ``` +If you install Manticore Search with the `manticore` package ([installation guide](https://manual.manticoresearch.com/Installation/Installation)), you already have MCL. That package provides `manticore-columnar-lib` and replaces the separate package, so do not install both. + Check that the libraries were loaded: `searchd -v` lists the columnar, secondary, knn and embeddings versions. Output from the `manticoresearch/manticore:29.9.0` image: ``` @@ -127,18 +129,18 @@ This produces `lib_manticore_columnar`, `lib_manticore_secondary` and `lib_manti ## Benchmarks -In 2022, the Manticore team published these comparisons of Manticore Search with columnar storage against Elasticsearch and ClickHouse on [db-benchmarks.com](https://db-benchmarks.com/). The test framework is [open source](https://github.com/db-benchmarks/db-benchmarks). The figures below are the "Relative query processing time" summaries from the published charts, which are based on geometric means of per-query response-time ratios. They reflect the software versions of that time. +The Manticore team publishes comparisons of Manticore Search with columnar storage against Elasticsearch and ClickHouse on [db-benchmarks.com](https://db-benchmarks.com/). The test framework is [open source](https://github.com/db-benchmarks/db-benchmarks). The figures below are the "Relative query processing time" values shown on the site, which are based on geometric means of per-query response-time ratios ("Fast avg" results, 110,000 MB RAM limit). They compare Manticore Search 27.1.5, Elasticsearch 9.4.3 and ClickHouse 26.6.1.1193. Each figure links to the live comparison. | Dataset | vs Elasticsearch | vs ClickHouse | |---|---|---| -| [10M Nginx logs](https://db-benchmarks.com/test-logs10m/) | [6.52x faster](https://db-benchmarks.com/test-logs10m/#elasticsearch-tuned-vs-manticore-search-columnar-storage) (tuned) | [1.40x faster](https://db-benchmarks.com/test-logs10m/#clickhouse-vs-manticore-search-columnar-storage) | -| [110M Hacker News comments](https://db-benchmarks.com/test-hn/) | [5.06x faster](https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-elasticsearch) (tuned) | [11.60x faster](https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-clickhouse) | -| [1.7B NYC taxi rides](https://db-benchmarks.com/test-taxi/) | [3.97x faster](https://db-benchmarks.com/test-taxi/#manticore-search-vs-elasticsearch) (tuned) | [1.81x faster](https://db-benchmarks.com/test-taxi/#manticore-search-vs-clickhouse) | +| [10M Nginx logs](https://db-benchmarks.com/test-logs10m/) | [2.79x faster](https://db-benchmarks.com/?cache=fast_avg&engines=elasticsearch_tuned_9.4.3,manticoresearch_columnar_27.1.5&tests=logs10m&memory=110000) (tuned) | [1.71x faster](https://db-benchmarks.com/?cache=fast_avg&engines=clickhouse_26.6.1.1193,manticoresearch_columnar_27.1.5&tests=logs10m&memory=110000) | +| [110M Hacker News comments](https://db-benchmarks.com/test-hn/) | [3.02x faster](https://db-benchmarks.com/?cache=fast_avg&engines=elasticsearch_tuned_9.4.3,manticoresearch_columnar_tuned_27.1.5&tests=hn&memory=110000) (tuned) | [8.99x faster](https://db-benchmarks.com/?cache=fast_avg&engines=clickhouse_26.6.1.1193,manticoresearch_columnar_tuned_27.1.5&tests=hn&memory=110000) | +| [1.7B NYC taxi rides](https://db-benchmarks.com/test-taxi/) | [2.02x faster](https://db-benchmarks.com/?cache=fast_avg&engines=elasticsearch_tuned_9.4.3,manticoresearch_columnar_27.1.5&tests=taxi&memory=110000) (tuned) | [3.16x faster](https://db-benchmarks.com/?cache=fast_avg&engines=clickhouse_26.6.1.1193,manticoresearch_columnar_27.1.5&tests=taxi&memory=110000) | -Most comparisons include queries where Manticore is slower; the per-query tables show them. In the 10M logs test, Elasticsearch timed out on 2 queries, which were excluded from its score. +In the Hacker News test, Manticore Search also runs with a tuned configuration. Every comparison includes queries where Manticore is slower; the per-query tables show them. In the 10M logs test, Elasticsearch timed out on 2 queries, which were excluded from its score.
-Charts +Charts from the original 2022 write-ups (older versions) **10M Nginx logs, compared with Elasticsearch (tuned)** @@ -170,7 +172,7 @@ Further reading on the Manticore blog: [About columnar storage](https://manticor ## Versions and releases -- The latest GitHub release is [14.2.1](https://github.com/manticoresoftware/columnar/releases/tag/14.2.1). It is the version in the current `manticore-columnar-lib` packages and in Manticore Search 29.9.0. +- The latest release is on [GitHub Releases](https://github.com/manticoresoftware/columnar/releases/latest). The `manticore` package and the official Docker image ship with a matching MCL version. - Pushes to `master` that change code get a version tag automatically from [`.github/workflows/auto_version.yml`](.github/workflows/auto_version.yml), so `master` can be ahead of the latest release. The `project()` version in `CMakeLists.txt` is a placeholder that the packaging workflow replaces. - Binary packages are built and published by the Manticore Search repository, not attached to GitHub releases here. - For release notes, see [GitHub Releases](https://github.com/manticoresoftware/columnar/releases) and the [Manticore Search changelog](https://manual.manticoresearch.com/Changelog), where MCL updates are listed. [`Changelog.md`](Changelog.md) in this repository covers only older versions.