Repository navigation
Conversation
ggml-org#27941 disabled -sm tensor for qwen4exp because test-llama-archs asserted on the Meta device once the fixture carried a PLE layer: GGML_ASSERT(ggml_backend_buffer_is_meta(tensor->buffer)) at ggml-backend-meta.cpp:476. With host-resident embeddings the PLE gather is a CPU node and hc_init (the REPEAT that fans the embedding out to the hc streams) was first reached through layer 0's PLE path, after that gather. ggml_backend_sched_split_graph pass 2 expands a device assignment upwards only until it meets a CPU node, so the REPEAT stayed on the CPU and the later reshape of hc_init inside the meta split viewed a host-resident node. Expanding hc_init right after it is built puts the REPEAT directly before the first device node, where pass 2 assigns it; the embedding reshape stays in the CPU split and is copied in as a split input, as in deepseek4.
The committed docs/ops/CPU.csv is out of sync with the current test-backend-ops suite: 11 ops with CPU support (COL2IM_1D, MUL_MAT_HADAMARD, SWIGLU_CLAMP, MUL_MAT_W4A4/W4A8, MUL_MAT_ID_W4A4/W4A8, DSV4_HC_COMB/PRE/POST, LIGHTNING_INDEXER) are missing entirely, and many other ops have fewer test cases than the suite generates now. docs/ops.md (which CI requires to match the CSVs) therefore understates CPU support. Regenerated with: test-backend-ops support -b CPU --output csv > docs/ops/CPU.csv scripts/create_ops_docs.py Note: ADD1 now reads unsupported on CPU because ggml_add1 is GGML_DEPRECATED and the suite no longer generates test cases for it; the CPU implementation itself is still present. Assisted-by: Xing
* OoD documenatation for llama-bench Signed-off-by: mairp <oec.valle.art@gmail.com> * Unset default: auto Signed-off-by: mairp <oec.valle.art@gmail.com> --------- Signed-off-by: mairp <oec.valle.art@gmail.com>
…8229) * bench : fix verbosity filter to show GGML_LOG_ERROR (ggml-org#28107) * bench: remove dead variables
* vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 The original tokenizer configs for PLaMo-2 and PLaMo-3 have `add_bos_token: true` and `add_eos_token: false` , but _set_vocab_plamo() did not write the BOS/EOS metadata. The PLAMO2 tokenizer path also ignored add_bos/add_eos during tokenization. Write the settings from tokenizer_config.json and honor them in the PLAMO2 tokenization path. GGUFs without these keys keep the previous behavior. * Update conversion/base.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
LLM-jp-4.1 uses the GPT-OSS format, but its tokenizer decodes a space after every special token and parallel tool calls are separated by <|end|>. The GPT-OSS handler rejects this output, so add a dedicated handler, selected by the chat_format=llm-jp-harmony-v1 declaration in the chat template. Assisted-by: Claude Fable 5.1
…org#29640) * BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS * AOCL-Blas : Add an AOCL-BLAS Quick Start and drop the fixed version path * AOCL-BLAS doc : Note on ZenDNN
…-io (ggml-org#29749) Assisted-by: Claude Co-authored-by: Pranesh Gonegandla <pgonegandla@nvidia.com>
…29792) Pinning to >= 3.4.3 is required to enable DeviceTopK, which was affected by a race condition NVIDIA/cccl#10627. We will relax this for future CTK versions which will bundle CCCL > 3.4.X (CTK 13.5 will bundle CCCL 3.5.0 for example)
* llama: fix qwen4exp * qwen4exp: keep kq_mask input the same shape
* Qwen4Exp: add MTP * remove has_state member, check via ctx_bufs being non-empty * consistent naming + less verbose comments * cont : clean-up recurrent memory * cont : clean-up comments --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* CUDA: Handle compute type for NVFP4 on cublass path Signed-off-by: ynankani <ynankani@nvidia.com> * Use BF16 compute type for quantized models if HW allows Signed-off-by: ynankani <ynankani@nvidia.com> * Set acc prec to bf16 for nvfp4 as it needs atleast bf16 range Signed-off-by: ynankani <ynankani@nvidia.com> * Update ggml/src/ggml-cuda/ggml-cuda.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * preserve op_params for per-expert matmul Signed-off-by: ynankani <ynankani@nvidia.com> --------- Signed-off-by: ynankani <ynankani@nvidia.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* metal : release temporary private transfer buffers Assisted-by: OpenAI Codex * metal : fix order and formatting --------- Co-authored-by: Niklas Wenzel <dev@nikwen.de>
* cuda : route sm70 to the Turing MMVQ nwarps table Volta (sm_70) has no MMVQ parameter table of its own and falls through to GENERIC, which launches K-quant batch-1 decode (ncols_dst == 1) at nwarps=4. sm_70 shares TURING's tuning: the K-quant vec_dot prefers nwarps=2 there. Route sm_70 to the existing MMVQ_PARAMETERS_TURING table in both the device and the host table selector. Measured on one Tesla V100 32GB PCIe (PG500-216, driver 580.178.04, CUDA 12.0.140) with Qwen3.8-27B Q4_K_M, tg128, interleaved A/B in 6 ABBA blocks with paired per-block deltas: +1.091 t/s = +3.17 % (t = +49.0, all six per-block deltas positive); perplexity bit-identical (6.3697 +/- 0.04066 both builds, wiki.test.raw). The patched build's K-quant mul_mat_vec_q kernels launch at nwarps=2 (cubin EIATTR_MAX_THREADS) while Q4_0/Q8_0 stay at nwarps=4, and the same measurement on the September master base gave +3.84 % (t = 85). The tuning originates from the V100-focused fork anyei/llamacpp-v100 (MIT), commit b912d1b1e, which carries a dedicated MMVQ_PARAMETERS_VOLTA table; a cubin-level comparison confirmed that routing sm_70 to the existing TURING table is equivalent for the K-quant batch-1 path this change affects, so this is the minimal 2-line form. anyei/llamacpp-v100@b912d1b1e Original-patch-by: anyei <angelyoelroblesmercedes@gmail.com> * Update ggml/src/ggml-cuda/mmvq.cu --------- Co-authored-by: tkittich <tkittich@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* convert : write Gemma embedding scale for DFlash drafts A DFlash draft shares the target's token embeddings. Gemma scales them by sqrt(hidden_size) in the forward pass, and the draft config does not state that scale, so the converted draft read unscaled embeddings. Take the scale from the target config when the draft config has none. Assisted-by: Claude * convert : check with get_model_architecture for gemma models
…ia DMA (ggml-org#29685) * hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA * hex-cpy: various fixes on top of the concat optimizations Removed CONCAT_DMA_MIN_ROW logic, it was broken with 64-bit DMA. While it's kinda silly to use DMA for tiny stuff if that tensor gets mapped to an extended buffer the only way to read it is DMA. Added missing dma_queue_flush() calls. Added additional guards for conditions we don't support. --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* tests : simplify function signature * llama : clamp kpool re-pool bound to existing pools The n_tokens/kpool + n_seqs_unq bound on n_new_g overshoots when a batch fills the whole cache: n_ctx tokens complete exactly n_ctx/kpool pools, so the +1 pads new_pool_idxs/new_pool_rep one entry past n_pool_real. Graph reserve only covers n_pool_real entries, so the first full-context decode builds bigger tensors than reserved and ggml-alloc demands a graph reallocation (abort under GGML_SCHED_DEBUG_REALLOC=1). Clamp the bound to n_pool_real: a ubatch can never mark more pools than the cache holds, and reserve's n_pool_max already covers that. Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD * cont : cap to n_pool_max
…9808) * skill: note about adding model-specific CLI arguments * add testing instructions
github-actions
Bot
added
documentation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| fix(fixer) { | ||
| // Add a second newline so the two siblings are separated by a | ||
| // blank line, keeping the trailing indentation. | ||
| return fixer.replaceText(textNode, textNode.value.replace(/\n/, '\n\n')); |
| const int leaf_backend_id = tensor_backend_id(leaf); | ||
| GGML_ASSERT(leaf_backend_id != -1); // all leafs should be assigned by now | ||
|
|
||
| if (tensor_id_copy(leaf_id, leaf_backend_id, 0) == NULL) { |
| } | ||
| ggml_set_input(tensor_copy); | ||
| ggml_set_output(tensor_copy); // prevent ggml-alloc from overwriting the tensor | ||
| tensor_id_copy(leaf_id, leaf_backend_id, c) = tensor_copy; |
| const int split_backend_id = split->backend_id; | ||
| ggml_backend_t split_backend = sched->backends[split_backend_id]; | ||
| ggml_backend_t input_backend = ggml_backend_sched_get_tensor_backend(sched, input); | ||
| struct ggml_tensor * input_cpy = tensor_copy(input, split_backend_id, sched->cur_copy); |
| wdata_src_b + i1n, 0, | ||
| wdata_kernel + i01*ne00*ne03 + i00*ne03, 0, 1); | ||
| } | ||
| dst_data[(i11*stride + i01)*ne0 + i10*stride + i00] += v; |
Split semicolon-joined statements (E702) in gguf_dspark_to_dflash.py and drop the unused typing.Any import (F401) in conversion/dspark.py.
The flake8 lint job runs flake8-no-print (NP100). Route the progress messages in gguf_dspark_to_dflash.py and the tiny DSpark/DFly test generators through a module logger, configured with a bare message format in __main__ so the output is unchanged.
khosravipasha
approved these changes
Oct 12, 2026
khosravipasha
left a comment
khosravipasha
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. This is not really possible to review so need to see what subset of functionatlies/hardware we want to test before mering.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges ggml-org master (5c8854f, 2026-10-11) into
prism. The fork was 941 commits behind, and upstream now carries the Prism Hadamard folded runtime from ggml-org#29600.Conflict decisions
llama_hadamard/hdmd). Fork-only pieces are re-applied on top: the target's transforms for drafters that borrow the target head and embeddings throughctx_other, and DSpark head borrowing. The MTP embedding inverse and the CPU_REPACK exclusion were already upstream.common_batch/llama_processAPI and the fused DFlash encoder. The fork's DSpark, prompt lookup (LLAMA_DFLASH_LOOKUP), beam selection and MTP deferred catch-up are ported over. DFly runs through the fused injection.Validation
Baseline is
prismat e8fd209 built the same way.dup=1cases, then an illegal memory access; unmodified upstream master does the sameprismbaseline (upstream master passes), so an existing fork issueNot covered: SYCL is not compiled, Vulkan and HIP are not built or tested, and the CUDA tg128 change was not compared against upstream master.