Repository navigation
Prism b10683 Ganglion Helios port multi slot concurrency speedup from kernel reuse - #243
Open
synchronic1 wants to merge 6 commits into
Open
synchronic1 wants to merge 6 commits into
synchronic1 wants to merge 6 commits into
Conversation
Cherry-pick of 918a9447b (llamAmpere decode-kernel port, based on upstream 465e49b) onto the PrismML fork so Bonsai 2 (PQ2_0/PTQ1_0) can run with the fused q8_0 K/V MMA attention, IQ4_XS/Q5_0 MMVQ reuse and Ampere launch table. Conflict resolution: - fattn-mma-f16.cuh: the fork predates upstream's sparse attention, so the use_sparse/indices parameters are dropped and only type_KV is added. - The fork also predates the smem swizzle; the q8_0 tile loader always uses the padded (nbatch + 4) layout the fork's kernels expect. - mmvq.cu: kept the fork's has_gate template parameter, GB10 prefetch and PTQ1_0 multi-column paths; the IQ4_XS/Q5_0 reuse branch sits beside them.
Adds vec_dot_pq2_0_q8_1_multi: each 32-element PQ2_0 chunk is loaded and expanded once and dotted with all ncols_dst (2..5) activation columns, instead of once per column. Per-column integer and float ops match vec_dot_pq2_0_q8_1, so results are bit-exact. Same gate as the IQ4_XS/Q5_0 reuse path (Ampere table, GGML_CUDA_MMVQ_REUSE=0 disables). Targets Bonsai 2 27B (PQ2_0) decode with --parallel > 1 or speculative verification, where MMVQ runs at ncols_dst 2..5.
The Ampere table's 8 rows per block at ncols_dst 2..4 regressed Bonsai 2 27B (PQ2_0) two-stream decode on an RTX 3060: 50.8 t/s vs 58.4 t/s on the unmodified fork. calc_rows_per_block now takes the weight type; PQ2_0 at ncols_dst == 2 uses 4 rows (GGML_CUDA_MMVQ_AMPERE_PQ2_0_RPB_NCOLS2), other types and widths are unchanged. llama-batched-bench, pp512 tg128, f16 KV, clocks locked, mean of 2 passes: streams fork port port reuse=0 1 34.57 34.58 34.51 2 58.41 59.86 59.15 3 62.75 68.52 62.31 4 61.54 76.05 75.40 Adds GGML_BONSAI_PERF perf cases for the Bonsai 2 27B GEMV shapes.
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The fused path still reserves full f16 KV buffers, and the benchmark annotations are misplaced.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
What changed in this PR
Adds Ampere CUDA optimizations for multi-column quantized matrix-vector operations and fused Q8_0 flash-attention decoding.
Changes:
- Reuses decoded weights across MMVQ columns.
- Decodes Q8_0 K/V data directly in MMA kernels.
- Adds opt-in correctness/performance coverage and benchmark results.
The PR template's AI usage disclosure remains incomplete.
| File | Description |
|---|---|
tests/test-backend-ops.cpp |
Adds opt-in MMVQ, Q8_0 attention, and PQ2_0 benchmarks. |
ggml/src/ggml-cuda/vecdotq.cuh |
Adds interleaved IQ4_XS lookup helpers. |
ggml/src/ggml-cuda/template-instances/generate_cu_files.py |
Generates Q8_0 MMA instances. |
ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_1-ncols2_8.cu |
Instantiates the 1-column Q8_0 kernel. |
ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_2-ncols2_8.cu |
Instantiates the 2-column Q8_0 kernel. |
ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_4-ncols2_8.cu |
Instantiates the 4-column Q8_0 kernel. |
ggml/src/ggml-cuda/template-instances/fattn-mma-f16-instance-ncols1_8-ncols2_8.cu |
Instantiates the 8-column Q8_0 kernel. |
ggml/src/ggml-cuda/quantize.cu |
Adds IQ4_XS-specific activation swizzling. |
ggml/src/ggml-cuda/mmvq.cu |
Adds Ampere tuning and multi-column weight reuse. |
ggml/src/ggml-cuda/fattn.cu |
Dispatches eligible Q8_0 attention to the fused kernel. |
ggml/src/ggml-cuda/fattn-mma-f16.cuh |
Implements staged Q8_0 decoding in MMA attention. |
benches/prism-helios-ampere-decode-perf.svg |
Adds Ampere throughput results. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+2186
to
+2187
| const bool need_f16_K = false; | ||
| const bool need_f16_V = false; |
get_alloc_size reserved f16 K/V buffers even when the fused q8_0 KV decode path runs, leaving dead space. Factor the eligibility checks into a helper shared by the dispatcher and get_alloc_size so the reservation is skipped exactly when the fused path is taken.
ggml_cuda_flash_attn_ext_get_alloc_size() reserved f16 copies of K and V for every MMA/TILE flash-attn op, but the fused q8_0 decode path reads the quantized cache directly and never uses them. Move the fused-path checks into ggml_cuda_flash_attn_ext_mma_q8_0_eligible() and call it from both the dispatcher and the alloc-size function, so the reservation is dropped exactly when the fused kernel runs. For a Bonsai 2 27B attention layer (hd 256, 24/4 heads) at n_kv 65536 this removes 256 MiB of dead workspace per FA op for 2..5 token batches.
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Additional information
Requirements