Skip to content

vulkan: fuse the sign multiply into the Hadamard FWHT - #349

Open
elliotmasheal wants to merge 1 commit into
PrismML-Eng:prismfrom
elliotmasheal:vulkan/signed-fwht-fusion
Open

elliotmasheal wants to merge 1 commit into
PrismML-Eng:prismfrom
elliotmasheal:vulkan/signed-fwht-fusion

Conversation

@elliotmasheal

@elliotmasheal elliotmasheal commented Oct 11, 2026 •

Copy link
Copy Markdown

Overview

The Vulkan backend was missing the signed Hadamard fusion that Metal already
has (ggml_metal_op_can_fuse_fwht_signed). On the folded Bonsai graphs the sign
multiply ran as its own elementwise pass over the activation, so every Hadamard
transform paid for an extra write, an extra read, and an extra barrier between
the two dispatches.

This folds MUL(signs) -> RESHAPE -> MUL_MAT(hint=hadamard) into a single
signed FWHT dispatch. The new shader variant reads the sign vector as a third
buffer and applies it while it loads the row, before the transform.

  • vulkan-shaders/fwht.comp: FWHT_SIGNED variant, third buffer for the sign
    vector, broadcast over signs->ne[0].
  • vulkan-shaders-gen.cpp: register fwht_signed_f32.
  • ggml-vulkan.cpp: pipeline_fwht_signed_f32, ggml_vk_can_fuse_fwht_signed
    (subgraph {MUL, RESHAPE, MUL_MAT} plus the hadamard hint, type, contiguity
    and shape checks), the dispatch, and the GGML_OP_MUL branch.

Additional information

Device: AMD Radeon RX 6700S (RDNA2, gfx1032), 8 GB, AMD proprietary driver,
Windows 11. Model: Ternary-Bonsai-2-27B (qwen35, Hadamard block 1024).

Command: llama-bench -m <model> -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1

model step pp512 tg128
PTQ1_0 baseline 62.48 18.16
PTQ1_0 + PTQ1_0 prefill loader (separate change) 78.80 18.16
PTQ1_0 + this fusion 108.77 24.13
PQ2_0 + this fusion 115.95 22.15
PQ2_0-MTP + this fusion 116.22 22.01

Decode isolates this change: the prefill loader touches only prefill, so
tg128 18.16 -> 24.13 on PTQ1_0 is the fusion alone (PQ2_0 16.88 -> 22.15).
The prefill numbers are the combined run with the loader present; the fusion
also removes the per-transform MUL and barrier in prefill.

Correctness:

  • llama-perplexity -f README.md -c 512 -b 512 --chunks 6 -fa 1 -dev Vulkan1 -ngl 999
    gives 4.2521 +/- 0.29627 for both PTQ1_0 and PQ2_0, identical to the baseline.
  • test-backend-ops -b Vulkan1: 17788/17788 pass (the known
    GATED_DELTA_NET rows_mode cases are skipped as before).

Limits and notes:

  • The fusion needs the subgroup FWHT pipeline; the wide FWHT_SHMEM branch
    falls back to the unfused path, so correctness there is unchanged.
  • The fusion predicate is strict: the MUL output must feed only that RESHAPE.
    When it does not match, the graph falls back to the separate ops.
  • GGML_VK_PERF_LOGGER=1 shows the MUL bucket at ~1.6 ms and the FWHT bucket at
    ~4.8 ms per token, but the observed saving is ~13.9 ms/token. The win is not
    just the elementwise pass: it is also the removed intermediate buffer traffic
    and the barrier between the two dispatches.
  • Measured on one RDNA2 device only; other Vulkan devices need their own numbers.
  • This is rebased onto the current prism. The numbers above were measured on
    the parent revision 2459f68; the intervening commits do not touch the
    Hadamard/FWHT path.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - the implementation and the benchmark write-up were drafted with an AI coding assistant working from my measurements; I reviewed every line, built the branch, and ran the PPL and test-backend-ops gates on hardware.

MUL(signs) -> RESHAPE -> MUL_MAT(hadamard) now runs as one dispatch: the
signed FWHT variant applies the sign vector while it loads the row, so the
separate elementwise pass and the barrier that followed it are gone. Mirrors
the Metal signed FWHT fusion.

Ternary-Bonsai-2-27B on RX 6700S (RDNA2), -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1:
  PTQ1_0  tg128 18.16 -> 24.13
  PQ2_0   tg128 16.88 -> 22.15
PPL 4.2521 +/- 0.29627 and the full test-backend-ops run are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant