Repository navigation
ptq1_0 Vulkan matvec kernel ~6x slower than tq2_0 at n=1 (195 vs 1080 GFLOPS, RX 6600) #259
Description
Decode-shaped MUL_MAT (m=4096, n=1, k=14336) measured via test-backend-ops: ptq1_0 = 602 us (195 GFLOPS), tq2_0 = 109 us (1080), q4_0 = 92 us. Root cause in vulkan-shaders/ptq1_0.glsl: trit accessor uses a 3-way branch + serial base-3 recurrence (v=(v*3)&0xFF loop) per weight element, unhidden at n=1. Prefill (n=512) is fine (4.11 TFLOPS), so only matvec path affected. Fix: 256-entry byte→5-trits LUT, same shape as mul_mat_vec_tq2_0.comp. Correctness currently 28/28 OK, so it's purely a perf bug. Real-world impact: Bonsai-2 27B decode 3.3 tok/s, should be ~20.
Activity
khosravipasha commentedon Sep 24, 2026
🤖 Answering on behalf of the maintainers
Thanks for reporting this and including the kernel timings and correctness checks!
The current
prismbranch now includes the faster PTQ1_0 integer-dot matvec kernel through #238, which brought in the work from #188 along with Intel dispatch changes. Testing reported a substantial decode improvement on an RX 6750 XT, but we haven’t confirmed the result on your RX 6600. This is newer than the demo’s pinned release. The slower fallback path is also being addressed separately in #252.If you can retest with a Vulkan build of the current
prismbranch, the build commit, driver version, command, and updated timings would help confirm what remains on your setup.We’re transferring this issue to our llama.cpp fork, where the kernels live. That’s the right place for backend/kernel performance reports like this; Bonsai-demo is mainly for setup and launcher issues. Thanks again for helping identify the bottleneck!