Track latest ggml-org llama.cpp again (keep Kokoro / OmniVoice / TBQ)#44
Open
odilitime wants to merge 909 commits into
Open
Track latest ggml-org llama.cpp again (keep Kokoro / OmniVoice / TBQ)#44odilitime wants to merge 909 commits into
odilitime wants to merge 909 commits into
Conversation
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* CUDA: Improve performance via less synchronizations between token (#17795) * Adds CPU-to-CUDA copy capability to ggml_backend_cuda_cpy_tensor_async() * Adds function to relax sync requirements between input copies on supported backends (CUDA for now) * Exchanges synchronous copy with async copy function. * Adds macro guards to allow compilation in non-CUDA builds * Reworked backend detection in ggml-backend.cpp to avoid linking conflicts * Relax requirement of checks in async CUDA copies from backend and buffer type to just buffer type, to avoid linking issues * Minor cleanup * Makes opt-in to relax use of explicit syncs more general. Backends like vulkan which require a synchronization between HtoD copies and graph execution could also adopt this change now. * Reintroduces stricter check for CPU->CUDA backend async copy via GGML_DEVICE_TYPE_CPU. * Corrects initialization of ggml_backend_sync_mode in ggml_backend_sched_split initialization * Simplifies synchronizations to adhere to `saaasg` pattern. * Apply suggestion from @ggerganov (src->buffer to buf_src) Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Apply suggestion from @ggerganov (src->buffer to buf_src) v2 Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Apply suggestions from @JohannesGaessler code review Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Adds single-GPU synchronizations to multi-GPU settings to fix hip backend pipeline parallel bugs. * Scheduler Hardening: Exclude hip/MUSA from copy_from_host CPU split -> GPU split optimization * Scheduler Hardening: Re-adding original additional synchronizations for non-async backends * Adds disclaimer to hip/musa exclusion of copy_from_host. Highlights that it is out of precaution, but that no perf-impact is visible, and that it can be revisited separately anytime. --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Tests are generally prefixed with -test, so rename export-graph-ops accordingly. rpc-server is probably too generic a name for /usr/bin. Because it should work with any ggml application, it is renamed to ggml-rpc-server.
* [CUDA] Added a cudaMemcpy2DAsync fast path to ggml_cuda_cpy Add a CUDA ggml_cpy fast path for same-type, same-shape strided copies that are just 2D pitched block copies. When tensors are not fully contiguous but each row is contiguous, it now uses cudaMemcpy2DAsync instead of the slow element-wise scalar copy kernel. This fixes the GDN recurrent snapshot update with -np 4, where rollback slots are separated by cache stride gaps. * Add new tests that execute the new optimized strided copy path * Return unsupported for strided copy in OpenVINO, as new tests are failing
* opencl: rework FA kernel for f16 and f32
* opencl: flash-attention prefill prepass kernels
- flash_attn_kv_pad_f16 pads the tail KV tile to a BLOCK_N multiple
- flash_attn_mask_pad_f16 pads the matching mask tile
- flash_attn_blk_f16 classifies each KV tile per query block as
fully masked / mixed / fully unmasked, so
the main kernel can skip fully-masked tiles
and the mask lookup for fully-unmasked ones
* opencl: FA kernels for q4_0 and q8_0
* opencl: `set_rows` for f32 to q8_0/q4_0
* opencl: dequant kernels for q4_0 and q8_0
* opencl: add FA tile tuning table with override
* opencl: wire host side for FA
* opencl: q4_0 MoE tensors are also SOA'ed
* opencl: cosmetic fix
* opencl: refactor, also clarify some code paths in comments
* opencl: fix inifity for `-cl-finite-math-only`
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* server : reduce logs * cont : common * cont : spec * cont : CMN_ -> COM_
Expose the existing --offline flag to `llama download` so a script can run it to check whether a model is already cached and ready to be served without touching the network. Also fix a latent use-after-free in the URL-task on_done callback: first_path is block-scoped and was captured by reference, but invoked after the block ends. Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* spec: add DFlash v2 support * dflash: support sliding window attention per layer_types * docs: add dflash section --------- Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
* jinja: add --dump-prog for debugging * Update common/jinja/runtime.cpp Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> --------- Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Add minicpm5 tool call parser * Refactor MiniCPM5 PEG parser per review feedback * Fix jinja min/max API to match Jinja2 * modify by review * MiniCPM5: use autoparser for XML tool calls and fix grammar preserved-token triggers * MiniCPM5: fix streaming tool-arg placeholder and remove alt XML markers * skip min/max attribute tests in -py mode * test-jinja: use real expected output for min/max attribute tests * MiniCPM5: revert shared mapper and history fallbacks per review Drop streaming tool-arg placeholder workarounds from the generic PEG mapper and restore strict tool-call argument JSON parsing so MiniCPM5 support stays limited to autoparser/diff-analyzer changes. * chat : refactor minicpm5 back to dedicated parser * cont : simplify grammar * cont : refactor * cont : fixes * cont : rename template to openbmb-MiniCPM5-1B.jinja * cont : add message delimiters * cont : fix tests --------- Co-authored-by: zhangtao <zhangtao2@modelbest.cn> Co-authored-by: 张涛 <>
* dflash: refactor draft model conversion * apply fix for eagle3 convert
…sisted by claude(in debugging and tests) (#24727)" (#25098)
* jinja, chat: add --reasoning-preserve flag * correct help message
* convert: add dsv4 conversion * add basic setup * add llm_graph_input_dsv4 * add save-load state * add sinkhorn eps - correction by @fairydreaming * add rope fix * cleanup dead code * fix bugs * support pro model: added by @fairydreaming * remove redundant V cache * Chat template * remove debugging leftovers * Add mechanism for inlining templates based on architecture * s/deepseek-v4-flash/deepseek4/g * s/deepseek-v4-flash/deepseek4/g continued * enable graph reuse * enable FA * fix test llama archs * rename * compatibility with antirez ds4 GGUFs * simplified set_gguf_parameters() by calling super class method, replaced moe.score_func with expert_gating_func. * reserve worst-case kv-cache * revert max split inputs * address review comments * add padding to enable FA * pad only the final value of plan.n_kv to 256 * remove built-in cpp chat template * cont: remove cpp built-in template * rm outdated test * replace ggml_view_3d() with ggml_reshape_3d() Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * only support n_seq=1 for now * remove unused var * cont: remove unused var * use scale bias * use correct ptr for can_reuse * remove gen-chat-inline-templates.py * simplify graph reuse * cont: cleanup * remove unused inputs * enable partial checkpointing * add correct shape for kq_mask + set llama_model_n_swa to 0 for dsv4 * precompute source_idx + add comment about dummy write * support multi-seq * remove restored_trim_pos * use split_equal when possible * fix indent * address review comments * use LLM_KV * fix ci --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…tic (#25005) * vulkan: extract flops calculation into function * use flops instead of matmul src0 tensor size for submission threshold * use unsigned ints
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…e (#20793)" (#25138)
* HIP: keep MMQ for gfx900 MoE and Q8_0, use hipBLAS for dense K-quants Assisted-by: GitHub Copilot CLI * HIP: tighten conditional block to be explicitly for gfx900 * HIP: Further simplified gfx900 conditional block * removed unnecessary comment
* vulkan: roll bk loop in matmul for asahi linux * vulkan: fix inline comment * vulkan: revert BK-loop unroll change * vulkan: edit spirv directly for asahi roll bk loop * vulkan: remove trailing whitespace at the end of comments
* CUDA: fix Gemma E4B MTP FlashAttention * remove unused template declaration
…clamp + stride) (#25103)
Eliza main is a 51-commit orphan history (no merge-base with ggml-org). Water-test branch starts at pin tip 86d86ed and brings over Kokoro, OmniVoice, eliza-shipped Metal kernels, and CPU TBQ/QJL/Polar sources. CMake still stock ggml — llama-server build proves upstream tip compiles; wiring voice/FFI + type enums is a follow-up commit.
github-actions
Bot
added
documentation
Unrelated histories block GitHub merge. Keep tree at ggml-org 86d86ed + Eliza overlay (kokoro/omnivoice/eliza-shipped/TBQ sources); mark origin/main merged without importing stale orphan-tree sources (e.g. dflash-draft).
Author
|
Deconflict update: histories were unrelated (~771 add/add). Content merge with Switched to Metal |
Author
|
Deconflict landed (
|
odilitime
changed the title
Skip TBQ/QJL/Polar parity gates until types land in ggml.h, keep WebGPU fixture generation for qwen35, disable OpenMP for wasm, link CUDA against toolkit stubs in Docker, ignore known HIP VGPR spills, and retry OpenVINO Windows builds after z-applocal races.
Hosted runners have no NVIDIA driver; GGML_CUDA_NO_VMM avoids linking libcuda.so.1 so the compile-only ubuntu-cuda-build job can finish.
odilitime
marked this pull request as ready for review
July 25, 2026 00:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why merge this
Eliza’s fork should track upstream llama.cpp again — new models, Metal/CUDA fixes, and server APIs — without throwing away Kokoro, OmniVoice, or TBQ/QJL/Polar.
Every week on a frozen tip makes rebases harder, blocks community patches, and forces agents/Mac builds to miss upstream reliability work. This PR is the water-test that unblocks that future: re-root onto a recent ggml-org tip while keeping Eliza product trees in-tree.
If this cannot land, feature work on top of a stale orphan history will keep getting more expensive. Landing it restores a shared base with ggml-org so the next PRs are normal diffs again.
Pain today
main@ad56033(Jul 3) sits on an orphan ~51-commit history with no merge-base against ggml-org/llama.cppmain-b10064-*do not mean the tree matches ggmlb10064git merge/ rebase against upstream is blocked; drift compounds every upstream releaseWhat you get
86d86ed4(b10064 line) as the new roottools/kokoro,tools/omnivoice,eliza-shipped/eliza-kernelsMetal, CPU TBQ/QJL/Polar sources, Eliza validation workflows/docsmainrecorded viamerge -s oursso GitHub can merge despite unrelated historiesllama-servercompiles on the tip todayThis is a re-root, not a three-way content squash of two full trees. That is intentional: content-merging orphan histories produced Frankenstein sources (e.g. stale
dflash-draftagainst new headers) that do not build.Why it is safe to merge
mainMERGEABLE)How to try (3 commands)
Sanity:
tools/kokoro,tools/omnivoice, andggml/src/ggml-metal/eliza-shippedare present.Proof
llama-serverbuild on tip3375f3a0c→version: 10117 (3375f3a0c)merge -s ours)Honest follow-up (not a reason to reject the re-root)
CMake is still stock ggml on this tip so
llama-serverproves the upstream base compiles. Next commit(s) should re-wire:elizainferenceCMake optionsThat work is easier on this tip than on the orphan history. Rejecting the re-root does not avoid that work — it only makes it harder.
Out of scope (deliberately)
Ask
Merge the re-root so Eliza
mainshares history with ggml-org again. Then land the small CMake/type follow-ups as normal PRs on a living tip — instead of fighting an orphan freeze forever.