Repository navigation
llama: GDN rollback by replay, one-graph MTP draft chain, K/V-only MTP prompt pass, sync trim (port of Torchit1/arc-b580) - #357
Merged
Conversation
…ec style) blk.<mtp>.nextn.draft_head (rows of output.weight for the K most frequent tokens plus all special tokens) and draft_head_ids (I32). When present, the MTP draft computes logits for those K rows only and scatters them into a full-vocab row of -inf with set_rows, so everything downstream is unchanged and the target model still verifies every draft. The head is listed under prism.hadamard.weight_names (same input-side rotation as output.weight). LLAMA_MTP_DRAFT_HEAD_OFF=1 uses the full head. Loader: GGML_OP_SET_ROWS placement check so the I32 id tensor lands in device memory. Bonsai 27B, K = 32768 (99.6% token coverage of a 4M-token code/prose corpus): the draft head reads 7.6x fewer bytes; fresh code temp 0 90.3 -> 94.8-95.8 t/s with identical output, temp 0.6 neutral within noise. K = 16384 is worse (92.1, 76 vs 81). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e75a994 on Torchit1/llama.cpp arc-b580)
…l accept, halve on rejection) A copy ends where the model stops but the context goes on, so with --spec-ngram-mod-n-max 256 a short copy drafted ~200 tokens past its end; at 48K context verifying those cost ~0.3 s each and made copying slower than writing new code. Drafts are now cut to n_cur (starts at 64, doubles after a fully accepted draft up to n_max, halves after a rejected one). B580, Bonsai 27B, MTP + ngram-mod, same outputs: 48K short copy 38.6 -> 57.3 t/s, 48K 1262-token copy 161.6 -> 172.9, 48K new code 78.5 -> 79.4, short rename 428 -> 411, edit 305 -> 299. LLAMA_ARG_SPEC_NGRAM_ADAPT=0 restores the fixed length; =N sets the starting length. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit e576a28 on Torchit1/llama.cpp arc-b580)
llama-mtp-export writes, per position, the trunk state the NextN (MTP) head consumes (h_nextn) and the trunk's greedy pick, for training / validating the draft head offline. common/speculative: LLAMA_MTP_DRAFT_LOG (top-3 candidates per draft step), LAB_SPEC_DRAFT_N_MAX / LAB_SPEC_NGRAM_START runtime knobs via GGML_SYCL_KNOBS_FILE. Used for the B580 self-distilled MTP head (+4.8% with MTP, outputs identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7c57726 on Torchit1/llama.cpp arc-b580)
…_ONLY) Independent implementation of the technique described by GRIMOIRE (written from a functional description and this fork's own code; no GRIMOIRE source read). A draft-context decode without output rows (the NextN head's prompt catch-up) only fills the head's KV cache, so graph_mtp now stops after the cache store: embedding (+ Hadamard inverse), hnorm/enorm, eh_proj, attn_norm, K/V projection, K norm, RoPE and cpy_k/cpy_v. No query, attention, output projection, FFN or head. The K/V ops are the full graph's on the same shapes and in the same expand order, so the cache rows are bit-identical. For batches of <= 8 rows the unused Q projection stays in the graph so the SYCL same-input mat-vec grouping (whose tile depends on the largest member) matches the full graph. Only masked nextn extraction qualifies (unmasked readers copy h for every row). Decodes with outputs build the unchanged graph. LLAMA_MTP_KV_ONLY=0 restores the full graph for every decode. Default on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 595d554 on Torchit1/llama.cpp arc-b580)
…MTP_CHAIN) All MTP draft steps in one llama_decode: each step takes the previous step's in-graph argmax and head output row; ids read back once. A 36 MiB device copy of the draft-vocabulary token_embd rows (PTQ1_0) plus SYCL GET_ROWS for PTQ1_0 (same arithmetic as the CPU) keep the lookups on the GPU. On by default only with a trimmed draft head; LLAMA_MTP_CHAIN=1 forces it, =0 off; LLAMA_MTP_CHAIN_GPU_EMBD=0 uses host lookups. Independent implementation of the technique described by llamADALL (clean-room: written from a functional spec, without reading llamADALL's source). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit bcbe5a0 on Torchit1/llama.cpp arc-b580)
Independent implementation of the technique described by llamADALL (clean-room, from the spec and this fork's own code only). With MTP, each verify pass used to write K = n_rs_seq + 1 full DeltaNet states per layer. The new op GGML_OP_GATED_DELTA_NET_REPLAY keeps one base state (the oldest one a rollback can need) plus a small update record (delta, k, g) per later token. The next pass first re-applies the records of the tokens that were kept, then runs the new tokens. The replay repeats the step's own state update, so the state is bit-identical to the snapshot path. - ggml: new op appended at the end of the enum (GGML_OP_COUNT 102, RPC patch version bump); CPU reference. - SYCL: sequential replay kernel. The blocked kernel can emit records (template flag; the existing instantiations are unchanged). A fused cache write (CPY + SET_ROWS skipped) and the gather-skip are used. - llama: recurrent memory keeps the base in plane 0 and two record banks in plane 1; the tensor layout is unchanged. Partial rollback = fewer records to replay. State save/restore carries the pending records (s_trans = 2). The conv state keeps its snapshot planes. - On by default for a single sequence, qwen35, all recurrent layers on SYCL. LLAMA_GDN_REPLAY=0 restores the snapshot path. - test-backend-ops: GATED_DELTA_NET_REPLAY chain tests (replay == snapshot exactly, and vs CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e37b71a on Torchit1/llama.cpp arc-b580)
…0.2..-0.8% on the B580) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4798af1 on Torchit1/llama.cpp arc-b580)
Independent implementation of the technique described by llamADALL (follow-up). The chain graph was built, split and allocated on every draft because allow_reuse refused chain graphs, which cost more host time than the chain saved (-0.2..-0.8% on the B580, drafts identical). Now: - llama_context keeps a scheduler + graph result per chain batch shape (steps, rows) and swaps them in around process_ubatch, so a draft of a known shape only sets inputs and computes; regular decodes keep their own gf_res_prev. - allow_reuse accepts chain graphs with the same step count; the chain input's can_reuse also requires the same rows and the same per-step KV lengths (get_n_kv_prefix), which are baked into the step views. - per-step position and output-row inputs are packed into two tensors (two uploads per draft instead of 2 x steps). Same graph, same values: outputs unchanged. Still default off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d6435c6 on Torchit1/llama.cpp arc-b580)
…_TIMING) Both measured runs of the draft chain spend ~6.5 ms per draft call, chain or loop, against a ~3.4 ms weight-read floor. To find where it goes: - LLAMA_MTP_DRAFT_TIMING=1: host phases of every MTP draft() call (pre-draft flushes, batch building, llama_decode submission, GPU wait + readback), averaged and printed every 64 calls, for both the chain and the loop. - GGML_SYCL_OP_TIMING=1: GPU time per op type for draft-sized graphs (< 2000 nodes, every mat-mul <= 8 rows), draining the queue after every kernel group (serialised upper bound incl. one launch latency each), every 64 graphs. Both off by default; no change to what is computed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 60edb38 on Torchit1/llama.cpp arc-b580)
Replay mode keeps the base state in S plane 0 and the two record banks in plane 1; planes 2..n_rs_seq were allocated but never used (432 MiB on Bonsai at n_rs_seq = 4, 48 GDN layers x 3 planes x 3 MiB, independent of -c). - llama_memory_recurrent: the replay decision now runs before the tensors are created (layer selection via the same filter as the allocation), and S gets 2 * mem_size rows in replay mode. The conv state r_l keeps its 1 + n_rs_seq planes and rs_idx. - LLAMA_GDN_REPLAY=0 and every case where replay does not apply keep the full (1 + n_rs_seq)-plane S allocation, unchanged. - No offset changes: the graph, the SYCL fused cache write and the state save/restore only ever address plane 0 (row head) and plane 1 (row size + head), so the s_trans = 0/2 file layout is the same and files from cr-replay still load. - build_recurrent_attn asserts the 2-plane S shape in replay mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit bda090e on Torchit1/llama.cpp arc-b580)
…eight type The draft-size test counted the Hadamard transform mat-muls, whose src1 holds each row as 1024-wide blocks (5 per 5120 row, 17 per FFN row), so every draft graph failed the <= 8 rows check and nothing was timed. Only weight mat-muls are checked now. A one-time "op-timing: active" line confirms the hook engaged, and mat-muls are reported per weight type (Q6_K / Q8_0 / PTQ1_0 head weights, the Hadamard transforms apart). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 24424cd on Torchit1/llama.cpp arc-b580)
…ies (LLAMA_MTP_CHAIN_TIGHT bit 1) Feature B round 2, idea 1 (independent implementation of the technique described by llamADALL). The chain graph sliced its packed host inputs (positions, output rows, K/V indices, mask) with per-step views built lazily inside each step. The scheduler makes every view of a host input its own split input: ~20 blocking host->device copies per draft (5 per step), each behind a backend sync, before the device split starts. And the views landed between the step's compute nodes. Now each packed input gets one whole-tensor copy (a device memcpy) pinned to the head's device, all per-step slices are views of those copies, and the host token lookup, the copies and every slice are added to the graph before the first compute node. Each step's compute nodes then run uninterrupted, as in the single-step draft graph; the device split keeps ~9 inputs instead of ~24. Same values in the same kernels: drafts unchanged. LLAMA_MTP_CHAIN_TIGHT is a bit mask (default 1 here); 0 restores the previous layout exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0e1ba9c on Torchit1/llama.cpp arc-b580)
…ous (LLAMA_MTP_CHAIN_TIGHT bit 2) Feature B round 2, idea 2 (independent implementation of the technique described by llamADALL). Every chain step ran a CONT on its mask slice. A 1-row step's slice is one row of the mask and already contiguous, so flash attention reads it in place (the decode kernels take the mask row stride from nb[1]; with one row it is never used). Only step 0 with catch-up rows, narrower than the full mask (its KV length is a prefix), still needs the copy. Saves n-1 copy kernels per draft (n = 4: 3). Same mask values: drafts unchanged. Default LLAMA_MTP_CHAIN_TIGHT is now 3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit c9c5053 on Torchit1/llama.cpp arc-b580)
…_CHAIN_TIGHT bit 4) Feature B round 2, idea 3 (independent implementation of the technique described by llamADALL). Every chain step ran GET_ROWS(h_nextn, out_idx) to pick its output row. Only step 0 (catch-up rows + the anchor) has a choice; a 1-row step's row is already the output. Now only steps with more than one row gather, and the output-index input holds only their entries (none when step 0 has no catch-up rows: no out_idx input at all). Side effect, on purpose: without the gather, the draft head's input chain of steps 1.. is ADD -> RMS_NORM -> MUL(norm w) -> MUL(signs) -> RESHAPE -> MUL_MAT(Hadamard), which ggml_sycl_try_add_norm_rot folds into one kernel (residual add + head norm + rotation). The loop never takes that path for the head (its out_ids gather sits in between). The fold is documented as the same arithmetic in the same order (byte-identical in the trunk), but it is new for drafts: first suspect if drafts ever differ (bit 4 off isolates it). Default LLAMA_MTP_CHAIN_TIGHT is now 7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 9d3313b on Torchit1/llama.cpp arc-b580)
…decode (LLAMA_MTP_DRAFT_TIMING) Diagnostics for the question whether the cached chain graph is reused in steady-state decoding (the chain's llama_decode took 1.9 ms at short context and 3.3-3.5 ms at 48K/115K). With LLAMA_MTP_DRAFT_TIMING=1, every 64 chain decodes print to stderr: mtp-chain-decode: 64 chain decodes, graph reused R, rebuilt M (<reason> n, ...); per decode: reuse check or build+alloc X ms, set_inputs Y ms, compute submit Z ms Miss reasons come from llm_graph_result::reuse_miss_reason(): "new" (shape never built), "params" (allow_reuse), or the type name of the first input that refuses (attn_kv = mask n_kv, mtp_chain = a step's KV length). draft-timing "decode" minus X+Y+Z is the llama_decode time outside process_ubatch (batch split, slot search). No change without the env. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 505f7b8 on Torchit1/llama.cpp arc-b580)
…g upload to its cause Each ggml_backend_sycl_synchronize and blocking buffer_set_tensor is counted under a cause, printed per 64 graphs of >= 2000 nodes (the target's graphs, so ~ per MTP cycle) with the totals per such graph: - the scheduler tags its copies (new ggml-base hook ggml_sync_trace_set_cause): "sched: split input <name>" (computed inputs marked), "sched: backend switch"; - anything else is labelled with the first two named callers outside libggml (backtrace + dladdr), e.g. "llama_get_logits_ith <- common_sampler_sample". Lines: "sync-trace: N big graphs (+M smaller), per big graph: S synchronizes, U blocking uploads (B bytes)", then one line per cause. Off by default; costs a backtrace per sync when on (host timing is then not representative). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b13c1b9 on Torchit1/llama.cpp arc-b580)
…ast one (LLAMA_SYNC_TRIM bit 1) Every result getter (llama_get_logits_ith, the sampled-token getters, llama_get_embeddings_nextn_ith, ...) calls llama_context::synchronize, and an MTP cycle calls several per row: the server samples each verified row, then the MTP drafter reads the target's h_nextn row by row. Each is a full device wait on the one in-order queue every context shares. Only the first after a decode waits for this context's work; a later one can only wait for the other context's work queued in between (e.g. the draft graph), which this context does not need. Now the context tracks whether it queued anything (graph compute and the async output reads, decode/encode) since its last synchronize, and skips the device wait if not. Not with pipeline parallelism (its copy rotation runs in the sync). Results are unchanged: they were complete already. LLAMA_SYNC_TRIM is a new bit mask (default 3; 0 = upstream behaviour). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 60ef575 on Torchit1/llama.cpp arc-b580)
…YNC_TRIM bit 2) With a quantized KV cache every attention graph reads attn_inp_k_rot (256 x 256 f32 here) and attn_inp_v_rot (64 x 64) as graph inputs. They live in host memory, so each graph uploads them again: per graph 2 split inputs, i.e. 2 device synchronizes + 2 blocking copies (~272 KB), in the target's verify graph and in the MTP draft graph alike (upstream has a TODO for this). The cache now also allocates them once in its own device buffer (only when the cache has one non-host buffer type) and build_input_k_rot / v_rot return those tensors; set_input_k_rot / v_rot skip a non-input tensor. Same values (ggml_gen_hadamard, copied once): outputs identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 4ec75c3 on Torchit1/llama.cpp arc-b580)
…it 8) Feature B round 2 (independent implementation of the technique described by llamADALL). Two chain inputs were uploaded separately though they need not be: - the output-row index (step 0 with catch-up rows) now rides at the end of the position input: one upload, sliced on the device; - V is stored with the K indices: for a non-transposed V cache set_input_v_idxs writes exactly what set_input_k_idxs writes, so the V index input is left out of the graph (llm_graph_input_attn_kv::set_input now skips an unallocated V index tensor). Per chain decode -1..2 split inputs (sync + blocking copy each). Same values in the same kernels: drafts unchanged. Default LLAMA_MTP_CHAIN_TIGHT is now 15. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f3aa69d on Torchit1/llama.cpp arc-b580)
…/dlfcn/execinfo) is Linux-only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 135a766 on Torchit1/llama.cpp arc-b580)
…ines small as a macro) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b5a7db2 on Torchit1/llama.cpp arc-b580)
…A_MTP_CHAIN_TIGHT bit 16) Feature B, idea 4 (independent implementation of the technique described by llamADALL). The chain read its n per-step argmax ids back with n separate 4-byte ggml_backend_tensor_get_async calls. On this SYCL/Level Zero stack each small device->host read costs ~0.2 ms (measured: reading only step 0's id cut the draft call from 6.07 to 5.16 ms). Now, after the last step, the ids are concatenated into one I32 [n] tensor (n-1 tiny CONCAT kernels at the end of the graph, after every step's compute nodes), marked as the only output, and llama_context reads it with one async copy into mtp_chain_out. The per-step argmax tensors are no longer graph outputs. get_mtp_chain_ids, the draft log and the ids are unchanged (the same integers, copied). Default LLAMA_MTP_CHAIN_TIGHT is now 31. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b0376ae on Torchit1/llama.cpp arc-b580)
…L device LLAMA_SYNC_TRIM and LLAMA_MTP_KV_ONLY were on by default for every backend but were only measured on SYCL. They now default on only when a SYCL device is registered; both env vars still force either way. Other backends keep the previous behaviour unless asked.
The one-graph draft chain commit added PTQ1_0 to the supports_op type list and a bit-exact f32 get_rows case, but prism already had both labels, so the SYCL build failed (duplicate case value). Keep one label each; get_rows keeps prism's XMX-layout assert and uses the chain's f32 dequant.
bri-prism
marked this pull request as ready for review
October 13, 2026 22:53
khosravipasha
approved these changes
Oct 13, 2026
khosravipasha
left a comment
khosravipasha
left a comment
Collaborator
There was a problem hiding this comment.
🤖 Reviewing on behalf of the maintainers
Thanks for carrying this work over and checking the Metal fallback! Approving the direction, but please address these before merging:
llama_sycl_present()checks for any registered SYCL device, not the backend actually used by the model/context. On a mixed Intel/NVIDIA machine this can enable the sync-trim and MTP K/V-only defaults for a CUDA context too. Please gate those defaults on the actual backend in use, or leave them off by default until that is handled.- Please complete the pending SYCL replay operator tests and end-to-end MTP comparison with replay enabled/disabled before shipping replay enabled by default.
This is a source review, not independent hardware validation. Land #356 first.
Collaborator
|
🤖 Reviewing on behalf of the maintainers Additional targeted checks beyond the pending replay-on/off comparison:
The existing unit and Metal results are useful, but do not establish correctness of these SYCL runtime/state transitions. |
Collaborator
|
🤖 Reviewing on behalf of the maintainers Please choose the most useful subset of the additional testing suggestions above; no need to turn this into an exhaustive matrix. My priorities would be replay-on/off correctness through partial acceptance and rollback, plus verifying the actual-backend gating fix. Save/restore with pending replay state is the next useful check if that path is exposed. You can decide how much of the remaining opt-in chain and cache coverage to do now versus follow up on, and note what remains untested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The second half of Jesse Symons' Torchit1/llama.cpp
arc-b580work: speculative decoding and recurrent-state changes insrc/,common/and the ggml core. 22 of the 23 commits are his, cherry-picked with authorship kept and the source commit noted; the last one is ours (below). Stacked on #356, which this PR targets; review that first.GGML_OP_GATED_DELTA_NET_REPLAY: rollback by replay instead of per-token state snapshots. One base state plus small update records per token; the next pass re-applies the kept ones. Bit-identical to the snapshot path. New op appended at the end of the enum, CPU reference and SYCL kernel. On by default only for a single qwen35 sequence whose recurrent layers all run on SYCL;LLAMA_GDN_REPLAY=0turns it off. Every other backend keeps the snapshot path.LLAMA_MTP_CHAIN=1, off by default, needs a GGUF with a trimmed draft head), K/V-only prompt pass for the draft head (LLAMA_MTP_KV_ONLY, default on with SYCL), and thellama-mtp-exporttool.LLAMA_SYNC_TRIM, default 3 with SYCL, else 0).The replay and the one-graph chain are independent implementations of techniques described by llamADALL; the K/V-only pass is described by GRIMOIRE. No code from either.
Why
Fewer state writes per verify and less VRAM for the delta-net state, and fewer host round trips per draft step. On the source branch's B580 this is +12 to +15% for MTP + n-gram over v0.8.
Our commit
llama: default sync trim and the MTP K/V-only pass on only with a SYCL device. In the source branchLLAMA_SYNC_TRIMandLLAMA_MTP_KV_ONLYdefault on for every backend, but they were only measured on SYCL. They now default on only when a SYCL device is registered; both env vars still force either way, so CUDA and Metal keep their current behaviour until someone measures them.Testing
test-backend-ops -o GATED_DELTA_NET19/19,GATED_DELTA_NET_REPLAYnot supported on Metal as expected (snapshot path kept). MTP greedy on a small qwen35 model with an MTP head, againstprism: same speed; output matches whereprismitself is run-to-run stable (greedy MTP on Metal is not reproducible onprismeither).GGML_SYCL_PTQ1_T2=alland unset:test-backend-ops:GATED_DELTA_NET_REPLAY8/8,GATED_DELTA_NET46/46.llama-server -np 1 -fa on -c 4096, draft K=2, greedy, 3 prompts x 128 tokens, mean of 2 rounds:LLAMA_GDN_REPLAY=125.73 t/s vs=025.22 (+2.0%), same acceptance (218/322), output byte-identical 3/3. With q8_0 K/V: 25.61 vs 25.07 (+2.2%). Plain decode on the same build: 14.50 t/s, so MTP is 1.77x.-np 4the snapshot path is used, which-vconfirms ("GDN rollback by replay: off").prism(which diverges on a different prompt). The divergence is deterministic across rounds and the f16 vs q8_0 K/V arms differ, consistent with near-ties under batched verify; margins not measured.Credit
Jesse Symons (@Torchit1). Three of the source branch's commits came from this fork (#311-#313) and are already here, so they are not repeated.