Repository navigation
perf: NEON kernel for 5/6-bit widths — scalar path is ~4.6x slower than the 4-bit NEON path #40
Description
Activity
- added a commit that references this issue
on Aug 29, 2026 Progress: window-extraction kernel shipped (a8edd9c)
score_neon_wide4v2: one unaligned u64 window per 8 dims (replaces 8×unpack_code), L1 gather, NEON mul/add with per-lane bucket order bit-identical toscore_wide_scalar(bitwise-tested, 5/6-bit × dims {128,384}).E2E (real, d=768, n=2000): 5-bit 4.27 → 3.23 ms/q (−24%).
Honest ceiling analysis for the remaining gap (3.23 vs 0.89 ms/q)
Per-dim cost now: vectorized mul/add (~same as 4-bit path) + scalar extraction & gather (shift/and + L1 load). The 4-bit path extracts with 2 NEON ops per 16 dims and gathers via
vqtbl4(64-byte table = 16 f32 entries).The vector gather for 5/6-bit cannot use
vqtbldirectly: tables are 128/256 bytes andvqtbladdressing caps at 64 bytes. The standard workaround (range-split gather:vqtbl4on low nibble +vqtbl4on high bit as select +vbsl) costs ~2–3× the 4-bit gather per dim.Projected split-layout ceiling (separate low-4-bit + high-1/2-bit streams, vector extraction, split gather): ≈ 2.2–2.7 ms/q — i.e. 5/6-bit can likely never beat the 4-bit NEON path, and at best approaches the existing residual mode (1.76 ms/q @ r@10 0.984) which needs zero new machinery.
Recommendation
- Users needing recall ≥0.98 AND <2 ms/q today: residual mode.
- Split-layout (specified above) is the only remaining lever for 5/6-bit latency; expected payoff is bounded (~1.8–3.2 ms/q → ~2.2–2.7 ms/q at best). Worth it only if a workload is pinned on plain-mode latency AND cannot rebuild as residual.
- AVX2 wide kernel: same analysis applies; scalar-wide remains correct and bit-identical on x86_64.
- added a commit that references this issue
on Aug 29, 2026 v2 kernel landed in develop (merged via #41, eb26d67)
score_neon_wide4(u64-window extraction + L1 gather + NEON mul/add, bitwise-tested vs scalar) is in develop. 5-bit: 4.27 → 3.23 ms/q (−24%).Remaining scope for this issue, in priority order:
- Split-layout codes (separate low-nibble + high-bit streams): the only lever toward ~2.2–2.7 ms/q. Requires format v1.6 or layout-in-header; bounded payoff per ceiling analysis above.
- AVX2 wide kernel for x86_64 hosts (currently scalar-wide, bit-identical, correct).
Keeping this open for the split-layout decision; no urgency — residual mode already covers the recall≥0.98 + <2 ms/q niche at 1.76 ms/q.
- added a commit that references this issue
on Aug 29, 2026 Closed: primary scope shipped, remaining scope declined with rationale
Shipped (merged via #41):
score_neon_wide4batch kernel for 5/6-bit — u64-window extraction + L1 gather + NEON mul/add, bitwise parity vs scalar (tested at 5/6-bit × dims {128,384}).- 5-bit scan latency: 4.27 → 3.21 ms/q (−24%). The original "~4.6x slower" gap is now 3.6x, with extraction (not SIMD math) as the documented bottleneck.
Declined, with analysis (see ceiling comment above):
- Split-layout codes (v1.6): projected only ~2.2–2.7 ms/q best case; requires a format change + migration; and the niche it targets (recall ≥0.98 + <2 ms/q) is already served by residual mode (1.76 ms/q @ r@10 0.984, shipped Residual quantization (TurboQuant-style second pass) — recall at same bit width #23/feat(core): residual quantization — opt-in recall mode, format v1.4 (#23) #38). Reopen with a workload profile that is pinned to plain-mode latency and cannot rebuild as residual.
- AVX2 wide kernel: same extraction bottleneck applies on x86_64; scalar-wide is correct and bit-identical. Not worth it before split-layout exists.
Final landscape is documented in docs/BENCHMARK.md (width matrix) — close as completed.
- added a commit that references this issue
on Aug 30, 2026 - added a commit that references this issue
on Sep 14, 2026
Found during #39 implementation (E2E bench, real EmbeddingGemma d=768, n=2000, 100 queries, release)
Recall/size goals of #39 are met, but 5/6-bit scoring currently runs the portable scalar unpack loop instead of the batch-4 NEON kernel, costing ~4.6x query latency vs 4-bit.
Scope
score_neon_wideon aarch64: unpack 5/6-bit codes to bytes (or 16-bit indices), LUT gather from 32/64-entry centroid tables, widen+accumulate — keep bit-identity withscore_wide_scalar(existing bitwise parity test pattern).score_neon4whenbits != 4.use_avx2 && bits != 4gate already exists).neon_matches_scalar_bitwise-style parity test extended to 5/6-bit.Notes