Repository navigation
vulkan: fuse the sign multiply into the Hadamard FWHT - #349
Open
elliotmasheal wants to merge 1 commit into
Open
elliotmasheal wants to merge 1 commit into
elliotmasheal wants to merge 1 commit into
Conversation
MUL(signs) -> RESHAPE -> MUL_MAT(hadamard) now runs as one dispatch: the signed FWHT variant applies the sign vector while it loads the row, so the separate elementwise pass and the barrier that followed it are gone. Mirrors the Metal signed FWHT fusion. Ternary-Bonsai-2-27B on RX 6700S (RDNA2), -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1: PTQ1_0 tg128 18.16 -> 24.13 PQ2_0 tg128 16.88 -> 22.15 PPL 4.2521 +/- 0.29627 and the full test-backend-ops run are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The Vulkan backend was missing the signed Hadamard fusion that Metal already
has (
ggml_metal_op_can_fuse_fwht_signed). On the folded Bonsai graphs the signmultiply ran as its own elementwise pass over the activation, so every Hadamard
transform paid for an extra write, an extra read, and an extra barrier between
the two dispatches.
This folds
MUL(signs) -> RESHAPE -> MUL_MAT(hint=hadamard)into a singlesigned FWHT dispatch. The new shader variant reads the sign vector as a third
buffer and applies it while it loads the row, before the transform.
vulkan-shaders/fwht.comp:FWHT_SIGNEDvariant, third buffer for the signvector, broadcast over
signs->ne[0].vulkan-shaders-gen.cpp: registerfwht_signed_f32.ggml-vulkan.cpp:pipeline_fwht_signed_f32,ggml_vk_can_fuse_fwht_signed(subgraph
{MUL, RESHAPE, MUL_MAT}plus the hadamard hint, type, contiguityand shape checks), the dispatch, and the
GGML_OP_MULbranch.Additional information
Device: AMD Radeon RX 6700S (RDNA2, gfx1032), 8 GB, AMD proprietary driver,
Windows 11. Model: Ternary-Bonsai-2-27B (qwen35, Hadamard block 1024).
Command:
llama-bench -m <model> -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1Decode isolates this change: the prefill loader touches only prefill, so
tg128 18.16 -> 24.13on PTQ1_0 is the fusion alone (PQ2_016.88 -> 22.15).The prefill numbers are the combined run with the loader present; the fusion
also removes the per-transform MUL and barrier in prefill.
Correctness:
llama-perplexity -f README.md -c 512 -b 512 --chunks 6 -fa 1 -dev Vulkan1 -ngl 999gives 4.2521 +/- 0.29627 for both PTQ1_0 and PQ2_0, identical to the baseline.
test-backend-ops -b Vulkan1: 17788/17788 pass (the knownGATED_DELTA_NET rows_modecases are skipped as before).Limits and notes:
FWHT_SHMEMbranchfalls back to the unfused path, so correctness there is unchanged.
When it does not match, the graph falls back to the separate ops.
GGML_VK_PERF_LOGGER=1shows the MUL bucket at ~1.6 ms and the FWHT bucket at~4.8 ms per token, but the observed saving is ~13.9 ms/token. The win is not
just the elementwise pass: it is also the removed intermediate buffer traffic
and the barrier between the two dispatches.
prism. The numbers above were measured onthe parent revision 2459f68; the intervening commits do not touch the
Hadamard/FWHT path.
Requirements