Repository navigation
llama: GDN rollback by replay, one-graph MTP draft chain, K/V-only MTP prompt pass, sync trim (port of Torchit1/arc-b580) - #358
Merged
Conversation
- SYCL: PTQ1_0 weights repacked to 2-bit at load and multiplied on XMX via the TernSYCL int8 x int2 DPAS kernels (vendored, BSD 3-Clause), activations quantized to int8 with round-to-nearest (GGML_SYCL_PTQ1_T2=ffn|all). - SYCL: decode attention for q4_0 KV serving 1-4 query tokens per launch; blocked gated delta-net; PTQ1_0 mat-vec kernels (single column, decode-once multi column, same-input merge, fused gate/up); narrow concat. - Vulkan: matching PTQ1_0 multi-column, decode attention, gated delta-net and coopmat work. - Speculative decoding: draft KV cache type (-ctkd/-ctvd) honoured by the MTP draft context, LLAMA_ARG_SPEC_DRAFT_UBATCH to shrink its compute buffer. - docs/bonsai-arc-b580.md: results at 128K on a B580, build and run instructions, switches, credits. Based on the bonsai-combo branch of professorpalmer/llama.cpp-ada-ternary (PrismML PR #221). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 753323e on Torchit1/llama.cpp arc-b580) Excluded from this port: .github docs ggml/src/ggml-vulkan Conflicts resolved against prism: fwht.cpp keeps prism's FWHT (#302) and adds this commit's signed variant beside it as ggml_sycl_op_fwht_signed; ggml_sycl_op_mul_mat_sycl takes fp16 for PQ2_0/PTQ1_0 as before (#278) plus this commit's large-batch rule; the T2 path runs before the PQ2_0 XMX path (#294), which still serves PQ2_0/PTQ1_0 when GGML_SYCL_PTQ1_T2 is unset.
PQ2_0 (2-bit codes 00=-1 01=0 10=+1, fp16 scale per 128) is the same size as the 2-bit XMX layout, so it is remapped in place. Code 3 (+2) has no 2-bit two's-complement form: the repack reports it and the matmul aborts with a message. test-backend-ops MUL_MAT type_a=pq2_0 with GGML_SYCL_PTQ1_T2=all: 84/84; ptq1 94/94. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5056bde on Torchit1/llama.cpp arc-b580)
The T2 kernels use 16-lane int8 x int2 DPAS builtins that only Xe2 and newer drivers provide; on an Arc 140T (Xe-LPG+) GGML_SYCL_PTQ1_T2=all failed with 'No kernel named ... was found'. Now checked once against the device architecture. GGML_SYCL_PTQ1_T2_ANYGPU=1 skips the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 731c1a8 on Torchit1/llama.cpp arc-b580)
The 2D block I/O builtins took the surface base as 'long', which is 32 bits in SYCL device code on Windows (the device follows the host's LLP64 data model), so the weight loads would have used truncated addresses. 'long long' is 64 bits on both; identical code on Linux. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2f0cf6e on Torchit1/llama.cpp arc-b580)
…default stays best) B580 pp1024 (ub 1024, all weights on XMX): 8x128/8x2 939 t/s (default), 8x128/16x1 949, others 480-857. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 244c4d4 on Torchit1/llama.cpp arc-b580)
… one-group kernel) The one-work-group top-k took ~550 us per MTP draft token on the B580 (248K-entry vocabulary). Now up to 256 groups per row with a pairwise local-memory merge, then one group merges the winners; ties keep the lower index. test-backend-ops TOP_K 325/325. 200K entries: k=1 241 -> 9.6 us, k=10 647 -> 71 us. Live settings: fresh +1-2%, rename/edit within noise (the speculative step is not purely GPU-bound). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit afe4c82 on Torchit1/llama.cpp arc-b580)
…_FA_DEC_DPAS) Q.K uses intel_sub_group_i8_i4_matrix_mad_k32 (IGC provides it on Xe2 though the public extension only lists i4_i4_k64): the raw 16-byte q4_0 block of a key XOR 0x88888888 is the B operand, one key per lane, against int8 Q with one RNE scale per row and block (q4_0 byte order absorbed by pairing dims l and l+16 in A). P.V computes O^T = V^T P^T on f16 DPAS so each lane dequantizes its own key's V block from 5 dword loads. Token chunks of 8 let one kernel serve 1..32 (or =N) query tokens, so n-gram verify batches stop falling back to oneDNN + a full-KV f16 conversion. Per layer (Bonsai 27B shape, real cache layout) old -> new, us: 16K 1 tok 266 -> 123, 4 tok 457 -> 244, 16 tok 2217 -> 776; 48K 1 tok 768 -> 433, 4 tok 1295 -> 671, 16 tok 6629 -> 1915. FLASH_ATTN_EXT decode cases incl. 12/16/17/32 tokens and the permuted layout: 57/57. GGML_SYCL_FA_DEC_DPAS_PROBE skips phases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e5a4d8a on Torchit1/llama.cpp arc-b580)
mkl_fa_online_softmax_chunk gave each query row to one work-item that walked the whole KV chunk twice, so neighbouring work-items read addresses chunk_size floats apart. Above the oneDNN KV cap this kernel was 72% of generation time (Bonsai 27B, 115K-token context: verify batches > 32 tokens and every new prompt). Now 256 work-items per row with coalesced reads and group reductions; same math. Bonsai 27B, live flags, 114.6K-token code context: new code 17.6 -> 44.0 t/s, edit 10.8 -> 27.5 t/s, identical output; short prompts unchanged. FLASH_ATTN_EXT with the MKL path forced (ONEDNN_MAX_KV=512): 82/84, the same two pre-existing prompt-batch failures as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a122b2b on Torchit1/llama.cpp arc-b580)
Quantized mul_mat with 33+ columns fell through to dequantize-all-of-src0 + GEMM per call. For speculative-verify sized batches that is ruinous: Gemma 4 12B q4_0 with n-gram drafts of 33-64 tokens spent ~94% of GPU time in dequantize_row_q4_0_sycl_reorder. Default stays 4 x MMVQ_MAX_BATCH_SIZE (32); with 128, Gemma 4 12B + QAT assistant (ngram n-max 64): rename 79 -> 186 t/s, edit 57 -> 136. Bonsai 27B unchanged within noise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e8bd534 on Torchit1/llama.cpp arc-b580)
Reported from an Arc 140T and reproduced on a B580: FLASH_ATTN_EXT with quantized K/V in the permuted layout returned ERR ~1.2 for every KV length from 1024 up (ggml-org#27769). The failing test builds the cache view with a gap between tokens (2x the heads allocated, half viewed); the oneDNN SDPA and MKL dequant paths take the token stride from the head count. llama.cpp's own cache view is always packed, so real models were unaffected (Bonsai 27B perplexity at 4K: oneDNN 8.7273, MKL 8.7311, TILE 8.7330, and still 8.7273 with this guard). Interleaved K/V with a gap now falls back to TILE. FLASH_ATTN_EXT 4137/4137 (was 2 failing), plus a new q4_0 real-layout sweep (kv 1024..8192, nb 33/64/512) 29/29. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 226166a on Torchit1/llama.cpp arc-b580)
…ewhere GGML_SYCL_FA_DEC_DPAS uses the 16-lane int8 x int4 DPAS builtin that only Xe2 and newer provide; on Alchemist / Xe-LPG it would fail at the first launch (like the T2 path did before 9c25b48). Now checked once against the device architecture, sharing ggml_sycl_device_is_xe2() with the T2 gate; GGML_SYCL_FA_DEC_DPAS_ANYGPU=1 skips the check. B580 unchanged: 4-token verify at 48K 671 us (1295 without). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d441076 on Torchit1/llama.cpp arc-b580)
…, head 256 x GQA 2 sliding) The s8 x s4 DPAS decode kernel takes the head size as a template parameter (K read block by block, P.V subgroups own D/16 dims) and serves two more shapes when GGML_SYCL_FA_DEC_DPAS is on: Gemma 4's global layers (512, 1 KV x 16 Q heads, token chunks of 2) and sliding layers (256, 2 per KV head, chunks of 8). Splits target ~256 work-groups in all (a single KV head used to leave the GPU idle); combine step templated on D. Global-layer attention at 48K, us: 1 token 1114 (f16 TILE) -> 236, 4 tokens 2011 -> 635. Gemma 4 12B QAT + assistant, q4_0 KV, 128K context: short 92 -> 100 t/s new code; at 48K new code 47 -> 65, edit 44 -> 57 (f16 KV was 55 / 47). Bonsai shape unchanged (4-token verify at 48K 638 us). FLASH_ATTN_EXT: Gemma global 20/20, sliding 10/10, Bonsai 36/36. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c759ddb on Torchit1/llama.cpp arc-b580)
… GGML_SYCL_Q4_0_DPAS) Verify batches of 5..256 columns on q4_0 weights used MMVQ column chunks (weights re-read per 8 columns) or dequantize-everything + GEMM. The new kernel streams each weight row once for up to 64 columns (chunks of 64 beyond): one lane per weight row, B = the row's 16 q4_0 bytes XOR 0x88888888 through intel_sub_group_i8_i4_matrix_mad_k32, A = activations quantized once per call to int8 (per row and 32-block, elements l / l+16 paired). Xe2 only (ggml_sycl_device_is_xe2). GGML_SYCL_Q4_0_DPAS_MIN / _MAX (5 / 256). 15360x3840 q4_0: 8 columns 150 -> 81 us, 16 columns 292 -> 89 us. Gemma 4 12B QAT + assistant: rename 188 -> 275 t/s, edit 138 -> 168 (ngram n-max 96), new code unchanged, rename/edit outputs byte-identical. MUL_MAT q4_0 Gemma shapes 1..256 columns: 20/20. Also Gemma 4 q4_0 mat-mul perf cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 03594d5 on Torchit1/llama.cpp arc-b580)
…2_W8A8_MIN) Before the TernSYCL path went live, PTQ1_0 prompts ran on the W8A8 oneDNN GEMM (pp512 1013); GemmMT is better for verify-sized batches but slower on full prompt batches. New ggml_sycl_w8a8_mul_mat_t2 converts the repacked 2-bit layout to int8 (per-row scale, group ratios) and uses the same gemm_s8. The conversion is a fixed cost per call, so it only pays from ~1024 columns: Bonsai 27B pp1024 (ub 1024) 941 -> 1062, pp512 901 -> 818 (so MIN=1024 on the live config); 48.6K-token prompt read 67.7 -> 61.6 s (719 -> 789 t/s). PPL c4096 8.7273 -> 8.7359 (+0.1%, int8 rounding as the old W8A8 path). Off by default (0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a68382d on Torchit1/llama.cpp arc-b580)
…ve split count; PTQ1_0 T2 NSG knob fattn-dec (DPAS q4_0 path, D 256): single-token decode copies each 128-key K and V tile into SLM with overlapped 16-byte loads and reads lane = key from there; the split count now fills one wave of work-groups (Xe cores x work-groups per core / KV heads) instead of a fixed 64 per head. B580: 48K 1 token 433 -> 244 us, 4 tokens 640 -> 456 us; whole model plain decode +12% at 48K, +21% at 115K, same KLD and output. GGML_SYCL_FA_DEC_STAGE=0 and GGML_SYCL_FA_DEC_SPLITS=64 restore the old behaviour. ptq1-t2: GGML_SYCL_PTQ1_T2_NSG=4 (4 sub-groups for 2-8 row GEMV, +~1%, default off). Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 9cfffcb on Torchit1/llama.cpp arc-b580)
…4_matrix_mad_k32 builtin The builtin is not in the public subgroup matrix extension list. On a Lunar Lake (Arc 140V) driver that does not provide it, loading the device image failed with "No device image found for external symbol _Z36intel_sub_group_i8_i4_matrix_mad_k32" and every model crashed at the first decode. xe2-dpas.hpp emits the dpas.s4.s8 instruction directly (as libxsmm/TernSYCL's xe2.hpp helpers), so the symbol is gone from libggml-sycl. Used by the q4_0 decode attention and the q4_0 small-batch GEMM. B580: FA 4167/4167, MUL_MAT q4_0 (GGML_SYCL_Q4_0_DPAS=1) 59/59; whole model output byte-identical to the builtin at short / 48K / 115K context, plain and MTP. Single-token attention kernel ~6% slower (48K 244 -> 259 us), whole model within 1-2%. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit f31a4f6 on Torchit1/llama.cpp arc-b580)
… code split per kernel - ternsycl/: libxsmm/TernSYCL main (11484da) + the long long address fix of TernSYCL#3. Inline vISA only, so the undocumented builtins the old copy declared (intel_sub_group_i8_i2_matrix_mad_k32, __builtin_IB_* 2D block I/O) are gone; it also drops our three JIT workarounds. The per-32 activation scale variant (off by default) is removed. - ptq1-t2.cpp is compiled with -fp-model=precise: under icpx's default fast model the TernSYCL GemmMT returns NaN (upstream's own driver fails validation built that way). - fattn-dec: the f16 P.V DPAS also goes through inline vISA (xe2-dpas.hpp dpas_hf_r8). - CMake: -fsycl-device-code-split=per_kernel (GGML_SYCL_DEVICE_CODE_SPLIT_PER_KERNEL, default ON): the Xe2 XMX kernels are only JIT-compiled when launched, so GPUs without DPAS (Xe-LP, Gen9) don't compile them at all. - test-backend-ops: FA cases for 64-256 query tokens and ~115K history. B580: libggml-sycl has 0 matrix_mad / __builtin_IB references (was f16, i8_i2, 9 IB). FA 4167/4167, MUL_MAT q4_0 59/59, ptq1_0 T2 94/94. KLD (ub 1 / 4 / 512) identical to the previous build. Whole model output identical; MTP 48K 73.6 t/s, 115K 59.6 (was 69.6 / 57.2), plain unchanged. Startup unchanged (first request 4 s). Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 2e8a0e6 on Torchit1/llama.cpp arc-b580)
Long n-gram verify batches now stay on the q4_0 DPAS kernel instead of oneDNN's path, which converts the whole KV cache to f16 each call. B580 per layer: 48K 64 tokens 28.8 -> 4.9 ms, 115K 256 tokens 253 -> 44 ms. KLD (ub 64-256) 0.012699 vs 0.012800, same top p 92.60 vs 92.89 %, equal within error. Whole model MTP: 48K 73.2 -> 76.1 t/s, 115K 60.3 -> 65.1. GGML_SYCL_FA_DEC_DPAS=32 restores the old cap. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 064461a on Torchit1/llama.cpp arc-b580)
MTP with 4 drafts verifies 5 tokens per step, which ran the 8-token DPAS kernel with 3 empty tokens (48 rows of Q.K / softmax for 30). A 5-token chunk (128-key tiles) cuts the kernel from 637 to 502 us per layer at 48K (215 -> 175 us at 16K); whole model MTP 48K 77.4 -> 78.1 t/s, 115K 65.7 -> 67.1, output identical. GGML_SYCL_FA_DEC_NQ5=0 turns it off (=2 SLM-staged and =3 256-key tiles were slower). test-backend-ops: 5/6-token real-layout cases. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 081d868 on Torchit1/llama.cpp arc-b580)
…MA_ARG_SPEC_NGRAM_FRESH - ggml_sycl_knob(name, def): the attention (DPAS cap, staging, 5-token chunks, split count, probe) and PTQ1_0 T2 (tile, sub-groups, GEMM off) switches are read through it. With GGML_SYCL_KNOBS_FILE set, the NAME=VALUE file is re-read at the start of a graph compute when it changes, so one server can A/B variants. Without it, each knob is read from the environment once, as before. - LLAMA_ARG_SPEC_NGRAM_FRESH=1: the n-gram drafter forgets earlier requests at the start of each one, so repeated benchmark prompts don't replay earlier answers as drafts. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit 9db9408 on Torchit1/llama.cpp arc-b580) Excluded from this port: common
Back-to-back T2 mat-vecs on one tensor reuse its int8 activations (GGML_SYCL_PTQ1_T2_QCACHE), and the signed 1024-point FWHT that feeds them runs 128 lanes per block and writes Aq/SA itself (GGML_SYCL_PTQ1_T2_FWHTQ). B580 Bonsai 27B plain decode +3.6-4.9%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 254dfb2 on Torchit1/llama.cpp arc-b580)
ADD + RMS_NORM + MUL(weight) [+ MUL(signs) + RESHAPE + Hadamard MUL_MAT] runs as one work-group per token (GGML_SYCL_PTQ1_T2_NORMQ). Outputs only used inside the chain are not written; across tokens, outputs must not partly overlap inputs. B580 Bonsai 27B plain decode +1.9-2.6%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4d50f84 on Torchit1/llama.cpp arc-b580)
Upstream fixed the GemmMT row-scale broadcast that returned NaN under icpx's default -fp-model=fast (libxsmm/TernSYCL#1). B580: MUL_MAT ptq1_0/pq2_0 178/178 with fast math, KLD unchanged, decode speed unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 546f94c on Torchit1/llama.cpp arc-b580)
…e non-weight cost) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2dbd839 on Torchit1/llama.cpp arc-b580)
…, W8A8 threshold as a runtime knob; test shapes GGML_SYCL_HOST_TIMING=1 splits decode graphs into between / enqueue / wait host time plus per-op enqueue cost, without a tracer. test-backend-ops: Bonsai attn k/v and output-head mat-vec shapes, FA nh=1,2 at 48K. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit ec71714 on Torchit1/llama.cpp arc-b580)
…NV_CHAIN) CONCAT + conv-cache CPY + SSM_CONV + SILU + L2_NORM in one kernel per layer (one work-group per 128-channel head). Outputs are written at the CONCAT's position, so nodes in between must not touch them (checked); an output may sit exactly on the qkv input (same element per channel), and an L2 output on top of the old conv state falls back to its own L2_NORM kernel. Also GGML_SYCL_FAST_SET_TENSOR (opt-in pinned input uploads). B580 Bonsai 27B plain decode +1.3-1.8%, KLD ub1 unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b56951e on Torchit1/llama.cpp arc-b580)
bf16 alpha + beta mat-vecs (2 x 48 rows, same input, up to 8 tokens) with the dt-bias add, softplus, ssm_a multiply and sigmoid in one kernel: 9 graph nodes / 7 launches per layer -> 1 (2 when an output aliases the input and goes through a scratch buffer). GGML_SYCL_DN_GATES_DEBUG=1 prints why a block is not fused. B580 Bonsai 27B plain decode +0.5-1.4%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 38c6be5 on Torchit1/llama.cpp arc-b580)
q+k+v, GDN qkv+z and FFN gate+up (<= 8 tokens) share one quantization and one GEMV launch: TernSYCL's Gemv body now takes its work-group indices (run()), and a wrapper maps each group to its matrix through a single call site (three inlined copies spilled: 34.6 vs 46.3 t/s). Grouping and output staging as the PTQ1_0 merge. B580 Bonsai 27B plain decode +1.8-2.0%, KLD unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 31ae86e on Torchit1/llama.cpp arc-b580)
SWIGLU -> MUL(signs) -> RESHAPE -> Hadamard MUL_MAT: the rotate/quantize kernel computes silu(gate) * up on its load (GGML_SYCL_GLU_ROT). ADD + RMS_NORM + MUL whose rotation comes ~9 nodes later (DeltaNet layers): the rotation is computed with the norm when no node in between touches its output (GGML_SYCL_NORM_ROT_FAR). Same arithmetic, byte-identical outputs. B580 Bonsai 27B plain +0.3-0.8%, MTP rename +2%, edit +1.1%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit cf4b155 on Torchit1/llama.cpp arc-b580)
…EMV_MAX knob
arctune sweep on the B580: tile {8,128,2,4} is 7-11% faster per pass for 12-32 rows
(n-gram verify batches), the default {8,128,8,2} from 48 rows up. GGML_SYCL_PTQ1_T2_TILE
now forces one tile for all sizes; GGML_SYCL_PTQ1_T2_TILE_SMALL_MAX (default 32) picks the
crossover. GGML_SYCL_PTQ1_T2_GEMV_MAX (default 8) makes the GEMV/GemmMT switch a knob;
the sweep confirmed 8. Outputs and KLD identical; MTP rename/edit +0.5%, 48K copy +1.3%.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a70fe64 on Torchit1/llama.cpp arc-b580)
…_ONLY) Independent implementation of the technique described by GRIMOIRE (written from a functional description and this fork's own code; no GRIMOIRE source read). A draft-context decode without output rows (the NextN head's prompt catch-up) only fills the head's KV cache, so graph_mtp now stops after the cache store: embedding (+ Hadamard inverse), hnorm/enorm, eh_proj, attn_norm, K/V projection, K norm, RoPE and cpy_k/cpy_v. No query, attention, output projection, FFN or head. The K/V ops are the full graph's on the same shapes and in the same expand order, so the cache rows are bit-identical. For batches of <= 8 rows the unused Q projection stays in the graph so the SYCL same-input mat-vec grouping (whose tile depends on the largest member) matches the full graph. Only masked nextn extraction qualifies (unmasked readers copy h for every row). Decodes with outputs build the unchanged graph. LLAMA_MTP_KV_ONLY=0 restores the full graph for every decode. Default on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 595d554 on Torchit1/llama.cpp arc-b580)
…MTP_CHAIN) All MTP draft steps in one llama_decode: each step takes the previous step's in-graph argmax and head output row; ids read back once. A 36 MiB device copy of the draft-vocabulary token_embd rows (PTQ1_0) plus SYCL GET_ROWS for PTQ1_0 (same arithmetic as the CPU) keep the lookups on the GPU. On by default only with a trimmed draft head; LLAMA_MTP_CHAIN=1 forces it, =0 off; LLAMA_MTP_CHAIN_GPU_EMBD=0 uses host lookups. Independent implementation of the technique described by llamADALL (clean-room: written from a functional spec, without reading llamADALL's source). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit bcbe5a0 on Torchit1/llama.cpp arc-b580)
Independent implementation of the technique described by llamADALL (clean-room, from the spec and this fork's own code only). With MTP, each verify pass used to write K = n_rs_seq + 1 full DeltaNet states per layer. The new op GGML_OP_GATED_DELTA_NET_REPLAY keeps one base state (the oldest one a rollback can need) plus a small update record (delta, k, g) per later token. The next pass first re-applies the records of the tokens that were kept, then runs the new tokens. The replay repeats the step's own state update, so the state is bit-identical to the snapshot path. - ggml: new op appended at the end of the enum (GGML_OP_COUNT 102, RPC patch version bump); CPU reference. - SYCL: sequential replay kernel. The blocked kernel can emit records (template flag; the existing instantiations are unchanged). A fused cache write (CPY + SET_ROWS skipped) and the gather-skip are used. - llama: recurrent memory keeps the base in plane 0 and two record banks in plane 1; the tensor layout is unchanged. Partial rollback = fewer records to replay. State save/restore carries the pending records (s_trans = 2). The conv state keeps its snapshot planes. - On by default for a single sequence, qwen35, all recurrent layers on SYCL. LLAMA_GDN_REPLAY=0 restores the snapshot path. - test-backend-ops: GATED_DELTA_NET_REPLAY chain tests (replay == snapshot exactly, and vs CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e37b71a on Torchit1/llama.cpp arc-b580)
…0.2..-0.8% on the B580) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4798af1 on Torchit1/llama.cpp arc-b580)
Independent implementation of the technique described by llamADALL (follow-up). The chain graph was built, split and allocated on every draft because allow_reuse refused chain graphs, which cost more host time than the chain saved (-0.2..-0.8% on the B580, drafts identical). Now: - llama_context keeps a scheduler + graph result per chain batch shape (steps, rows) and swaps them in around process_ubatch, so a draft of a known shape only sets inputs and computes; regular decodes keep their own gf_res_prev. - allow_reuse accepts chain graphs with the same step count; the chain input's can_reuse also requires the same rows and the same per-step KV lengths (get_n_kv_prefix), which are baked into the step views. - per-step position and output-row inputs are packed into two tensors (two uploads per draft instead of 2 x steps). Same graph, same values: outputs unchanged. Still default off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d6435c6 on Torchit1/llama.cpp arc-b580)
…_TIMING) Both measured runs of the draft chain spend ~6.5 ms per draft call, chain or loop, against a ~3.4 ms weight-read floor. To find where it goes: - LLAMA_MTP_DRAFT_TIMING=1: host phases of every MTP draft() call (pre-draft flushes, batch building, llama_decode submission, GPU wait + readback), averaged and printed every 64 calls, for both the chain and the loop. - GGML_SYCL_OP_TIMING=1: GPU time per op type for draft-sized graphs (< 2000 nodes, every mat-mul <= 8 rows), draining the queue after every kernel group (serialised upper bound incl. one launch latency each), every 64 graphs. Both off by default; no change to what is computed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 60edb38 on Torchit1/llama.cpp arc-b580)
Replay mode keeps the base state in S plane 0 and the two record banks in plane 1; planes 2..n_rs_seq were allocated but never used (432 MiB on Bonsai at n_rs_seq = 4, 48 GDN layers x 3 planes x 3 MiB, independent of -c). - llama_memory_recurrent: the replay decision now runs before the tensors are created (layer selection via the same filter as the allocation), and S gets 2 * mem_size rows in replay mode. The conv state r_l keeps its 1 + n_rs_seq planes and rs_idx. - LLAMA_GDN_REPLAY=0 and every case where replay does not apply keep the full (1 + n_rs_seq)-plane S allocation, unchanged. - No offset changes: the graph, the SYCL fused cache write and the state save/restore only ever address plane 0 (row head) and plane 1 (row size + head), so the s_trans = 0/2 file layout is the same and files from cr-replay still load. - build_recurrent_attn asserts the 2-plane S shape in replay mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit bda090e on Torchit1/llama.cpp arc-b580)
…eight type The draft-size test counted the Hadamard transform mat-muls, whose src1 holds each row as 1024-wide blocks (5 per 5120 row, 17 per FFN row), so every draft graph failed the <= 8 rows check and nothing was timed. Only weight mat-muls are checked now. A one-time "op-timing: active" line confirms the hook engaged, and mat-muls are reported per weight type (Q6_K / Q8_0 / PTQ1_0 head weights, the Hadamard transforms apart). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 24424cd on Torchit1/llama.cpp arc-b580)
…ies (LLAMA_MTP_CHAIN_TIGHT bit 1) Feature B round 2, idea 1 (independent implementation of the technique described by llamADALL). The chain graph sliced its packed host inputs (positions, output rows, K/V indices, mask) with per-step views built lazily inside each step. The scheduler makes every view of a host input its own split input: ~20 blocking host->device copies per draft (5 per step), each behind a backend sync, before the device split starts. And the views landed between the step's compute nodes. Now each packed input gets one whole-tensor copy (a device memcpy) pinned to the head's device, all per-step slices are views of those copies, and the host token lookup, the copies and every slice are added to the graph before the first compute node. Each step's compute nodes then run uninterrupted, as in the single-step draft graph; the device split keeps ~9 inputs instead of ~24. Same values in the same kernels: drafts unchanged. LLAMA_MTP_CHAIN_TIGHT is a bit mask (default 1 here); 0 restores the previous layout exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0e1ba9c on Torchit1/llama.cpp arc-b580)
…ous (LLAMA_MTP_CHAIN_TIGHT bit 2) Feature B round 2, idea 2 (independent implementation of the technique described by llamADALL). Every chain step ran a CONT on its mask slice. A 1-row step's slice is one row of the mask and already contiguous, so flash attention reads it in place (the decode kernels take the mask row stride from nb[1]; with one row it is never used). Only step 0 with catch-up rows, narrower than the full mask (its KV length is a prefix), still needs the copy. Saves n-1 copy kernels per draft (n = 4: 3). Same mask values: drafts unchanged. Default LLAMA_MTP_CHAIN_TIGHT is now 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c9c5053 on Torchit1/llama.cpp arc-b580)
…_CHAIN_TIGHT bit 4) Feature B round 2, idea 3 (independent implementation of the technique described by llamADALL). Every chain step ran GET_ROWS(h_nextn, out_idx) to pick its output row. Only step 0 (catch-up rows + the anchor) has a choice; a 1-row step's row is already the output. Now only steps with more than one row gather, and the output-index input holds only their entries (none when step 0 has no catch-up rows: no out_idx input at all). Side effect, on purpose: without the gather, the draft head's input chain of steps 1.. is ADD -> RMS_NORM -> MUL(norm w) -> MUL(signs) -> RESHAPE -> MUL_MAT(Hadamard), which ggml_sycl_try_add_norm_rot folds into one kernel (residual add + head norm + rotation). The loop never takes that path for the head (its out_ids gather sits in between). The fold is documented as the same arithmetic in the same order (byte-identical in the trunk), but it is new for drafts: first suspect if drafts ever differ (bit 4 off isolates it). Default LLAMA_MTP_CHAIN_TIGHT is now 7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 9d3313b on Torchit1/llama.cpp arc-b580)
…decode (LLAMA_MTP_DRAFT_TIMING) Diagnostics for the question whether the cached chain graph is reused in steady-state decoding (the chain's llama_decode took 1.9 ms at short context and 3.3-3.5 ms at 48K/115K). With LLAMA_MTP_DRAFT_TIMING=1, every 64 chain decodes print to stderr: mtp-chain-decode: 64 chain decodes, graph reused R, rebuilt M (<reason> n, ...); per decode: reuse check or build+alloc X ms, set_inputs Y ms, compute submit Z ms Miss reasons come from llm_graph_result::reuse_miss_reason(): "new" (shape never built), "params" (allow_reuse), or the type name of the first input that refuses (attn_kv = mask n_kv, mtp_chain = a step's KV length). draft-timing "decode" minus X+Y+Z is the llama_decode time outside process_ubatch (batch split, slot search). No change without the env. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 505f7b8 on Torchit1/llama.cpp arc-b580)
…g upload to its cause Each ggml_backend_sycl_synchronize and blocking buffer_set_tensor is counted under a cause, printed per 64 graphs of >= 2000 nodes (the target's graphs, so ~ per MTP cycle) with the totals per such graph: - the scheduler tags its copies (new ggml-base hook ggml_sync_trace_set_cause): "sched: split input <name>" (computed inputs marked), "sched: backend switch"; - anything else is labelled with the first two named callers outside libggml (backtrace + dladdr), e.g. "llama_get_logits_ith <- common_sampler_sample". Lines: "sync-trace: N big graphs (+M smaller), per big graph: S synchronizes, U blocking uploads (B bytes)", then one line per cause. Off by default; costs a backtrace per sync when on (host timing is then not representative). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b13c1b9 on Torchit1/llama.cpp arc-b580)
…ast one (LLAMA_SYNC_TRIM bit 1) Every result getter (llama_get_logits_ith, the sampled-token getters, llama_get_embeddings_nextn_ith, ...) calls llama_context::synchronize, and an MTP cycle calls several per row: the server samples each verified row, then the MTP drafter reads the target's h_nextn row by row. Each is a full device wait on the one in-order queue every context shares. Only the first after a decode waits for this context's work; a later one can only wait for the other context's work queued in between (e.g. the draft graph), which this context does not need. Now the context tracks whether it queued anything (graph compute and the async output reads, decode/encode) since its last synchronize, and skips the device wait if not. Not with pipeline parallelism (its copy rotation runs in the sync). Results are unchanged: they were complete already. LLAMA_SYNC_TRIM is a new bit mask (default 3; 0 = upstream behaviour). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 60ef575 on Torchit1/llama.cpp arc-b580)
…YNC_TRIM bit 2) With a quantized KV cache every attention graph reads attn_inp_k_rot (256 x 256 f32 here) and attn_inp_v_rot (64 x 64) as graph inputs. They live in host memory, so each graph uploads them again: per graph 2 split inputs, i.e. 2 device synchronizes + 2 blocking copies (~272 KB), in the target's verify graph and in the MTP draft graph alike (upstream has a TODO for this). The cache now also allocates them once in its own device buffer (only when the cache has one non-host buffer type) and build_input_k_rot / v_rot return those tensors; set_input_k_rot / v_rot skip a non-input tensor. Same values (ggml_gen_hadamard, copied once): outputs identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4ec75c3 on Torchit1/llama.cpp arc-b580)
…it 8) Feature B round 2 (independent implementation of the technique described by llamADALL). Two chain inputs were uploaded separately though they need not be: - the output-row index (step 0 with catch-up rows) now rides at the end of the position input: one upload, sliced on the device; - V is stored with the K indices: for a non-transposed V cache set_input_v_idxs writes exactly what set_input_k_idxs writes, so the V index input is left out of the graph (llm_graph_input_attn_kv::set_input now skips an unallocated V index tensor). Per chain decode -1..2 split inputs (sync + blocking copy each). Same values in the same kernels: drafts unchanged. Default LLAMA_MTP_CHAIN_TIGHT is now 15. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f3aa69d on Torchit1/llama.cpp arc-b580)
…/dlfcn/execinfo) is Linux-only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 135a766 on Torchit1/llama.cpp arc-b580)
…ines small as a macro) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b5a7db2 on Torchit1/llama.cpp arc-b580)
…A_MTP_CHAIN_TIGHT bit 16) Feature B, idea 4 (independent implementation of the technique described by llamADALL). The chain read its n per-step argmax ids back with n separate 4-byte ggml_backend_tensor_get_async calls. On this SYCL/Level Zero stack each small device->host read costs ~0.2 ms (measured: reading only step 0's id cut the draft call from 6.07 to 5.16 ms). Now, after the last step, the ids are concatenated into one I32 [n] tensor (n-1 tiny CONCAT kernels at the end of the graph, after every step's compute nodes), marked as the only output, and llama_context reads it with one async copy into mtp_chain_out. The per-step argmax tensors are no longer graph outputs. get_mtp_chain_ids, the draft log and the ids are unchanged (the same integers, copied). Default LLAMA_MTP_CHAIN_TIGHT is now 31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b0376ae on Torchit1/llama.cpp arc-b580)
…L device LLAMA_SYNC_TRIM and LLAMA_MTP_KV_ONLY were on by default for every backend but were only measured on SYCL. They now default on only when a SYCL device is registered; both env vars still force either way. Other backends keep the previous behaviour unless asked.
The one-graph draft chain commit added PTQ1_0 to the supports_op type list and a bit-exact f32 get_rows case, but prism already had both labels, so the SYCL build failed (duplicate case value). Keep one label each; get_rows keeps prism's XMX-layout assert and uses the chain's f32 dequant.
# Conflicts: # ggml/src/ggml-sycl/dequantize.hpp # ggml/src/ggml-sycl/fattn-dec.cpp # ggml/src/ggml-sycl/gated_delta_net.cpp # ggml/src/ggml-sycl/gated_delta_net.hpp # ggml/src/ggml-sycl/ggml-sycl.cpp # ggml/src/ggml-sycl/ptq1-t2.cpp # ggml/src/ggml-sycl/ptq1-t2.hpp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same change as #357, which was approved and then squash-merged into #356's branch (
sycl/arc-b580-port) instead ofprism. This branch is #357's commits plus a merge ofprismafter #356 landed; the tree matches #357's merge result exactly.What
The second half of Jesse Symons' Torchit1/llama.cpp
arc-b580work: speculative decoding and recurrent-state changes insrc/,common/and the ggml core. 22 of the 23 commits are his, cherry-picked with authorship kept and the source commit noted; the last one is ours (below). Follows #356 (merged).GGML_OP_GATED_DELTA_NET_REPLAY: rollback by replay instead of per-token state snapshots. One base state plus small update records per token; the next pass re-applies the kept ones. Bit-identical to the snapshot path. New op appended at the end of the enum, CPU reference and SYCL kernel. On by default only for a single qwen35 sequence whose recurrent layers all run on SYCL;LLAMA_GDN_REPLAY=0turns it off. Every other backend keeps the snapshot path.LLAMA_MTP_CHAIN=1, off by default, needs a GGUF with a trimmed draft head), K/V-only prompt pass for the draft head (LLAMA_MTP_KV_ONLY, default on with SYCL), and thellama-mtp-exporttool.LLAMA_SYNC_TRIM, default 3 with SYCL, else 0).The replay and the one-graph chain are independent implementations of techniques described by llamADALL; the K/V-only pass is described by GRIMOIRE. No code from either.
Why
Fewer state writes per verify and less VRAM for the delta-net state, and fewer host round trips per draft step. On the source branch's B580 this is +12 to +15% for MTP + n-gram over v0.8.
Our commit
llama: default sync trim and the MTP K/V-only pass on only with a SYCL device. In the source branchLLAMA_SYNC_TRIMandLLAMA_MTP_KV_ONLYdefault on for every backend, but they were only measured on SYCL. They now default on only when a SYCL device is registered; both env vars still force either way, so CUDA and Metal keep their current behaviour until someone measures them.Testing
test-backend-ops -o GATED_DELTA_NET19/19,GATED_DELTA_NET_REPLAYnot supported on Metal as expected (snapshot path kept). MTP greedy on a small qwen35 model with an MTP head, againstprism: same speed; output matches whereprismitself is run-to-run stable (greedy MTP on Metal is not reproducible onprismeither).GGML_SYCL_PTQ1_T2=alland unset:test-backend-ops:GATED_DELTA_NET_REPLAY8/8,GATED_DELTA_NET46/46.llama-server -np 1 -fa on -c 4096, draft K=2, greedy, 3 prompts x 128 tokens, mean of 2 rounds:LLAMA_GDN_REPLAY=125.73 t/s vs=025.22 (+2.0%), same acceptance (218/322), output byte-identical 3/3. With q8_0 K/V: 25.61 vs 25.07 (+2.2%). Plain decode on the same build: 14.50 t/s, so MTP is 1.77x.-np 4the snapshot path is used, which-vconfirms ("GDN rollback by replay: off").prism(which diverges on a different prompt). The divergence is deterministic across rounds and the f16 vs q8_0 K/V arms differ, consistent with near-ties under batched verify; margins not measured.Credit
Jesse Symons (@Torchit1). Three of the source branch's commits came from this fork (#311-#313) and are already here, so they are not repeated.