Repository navigation
batch: land #229 - #266
Merged
Merged
batch: land #229#266
Conversation
With K3D_GPU_AGENT_GPUS=2 the launcher passed --gpus 2 to the fit for an OVMS agent too. The fit then sized a layer split over two cards and rendered a KV pool of two cards' worth (--cache_size=41 for the Gemma 4 int4 IR on the Xeon fixture) into a pod that holds one gpu.intel.com/xe. The fit now refuses a multi-GPU OVMS placement with "OVMS serves one GPU", and agent_fit passes --gpus only when the agent app is vLLM. When no placement can be tried at all, the refusal says no Intel GPU with device-local memory is free instead of echoing the engine note. Tests: OVMS with --gpus 2 is refused, the GPU cache_size stays within one card's budget, the no-free-GPU reason, and an agent manifest with K3D_GPU_AGENT_GPUS=2 under OVMS whose --cache_size fits one card.
The fit capped every engine's sliding-window layers at the window. OpenVINO GenAI 2026.4 has no sliding-window cache type (only KV_CACHE and LINEAR_ATTENTION_CACHE in continuous_batching/cache/cache_type.hpp), and one KV block spans every attention layer, so OVMS holds every token in those layers too. Gemma 4 costs 225280 B per token under OVMS, not 20480 B plus a 1024-token window; on the Xeon fixture the old fit set --cache_size=13 for 131072 x 5 sequences, which needs about 138 GiB. kv_profile and per_seq are now engine-aware through swa_tokens: OVMS counts sliding-window layers at the whole context, vLLM keeps the window cap, and llama.cpp counts its SWA cache of GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch 512), 256) cells per slot (src/llama-kv-cache-iswa.cpp at b11749), 1536 cells for Gemma 4's 1024 window. OVMS --cache_size is whole GiB and was floored, so the pool could hold fewer tokens than the fit counted. Whole-GiB pools (OVMS, and vLLM's CPU KV space) are now sized inside whole GiB of room and rounded up, on every guard rung too. Tests: per-engine SWA bytes for Gemma 4; the Gemma 4 OVMS cache_size holds context x sequences at 225280 B per token on a GPU, on the Xeon CPU and on the desktop, at every guard rung; the agent manifest renders such a pool; the Gemma QAT GGUF plan drops from 6 to 5 slots for the 1536-cell SWA cache.
llama-server b11749 keeps two host caches beside the KV cache that the fit did not count: a prompt cache of up to 8192 MiB (--cache-ram) and up to 32 context checkpoints per slot (-ctxcp), each a host copy of the slot's SWA cache and recurrent state. For the Gemma QAT GGUF on the desktop CPU plan (5 slots at 262144, KV filling the room) that is 8 GiB plus about 47 GiB of checkpoints over a 45Gi pod limit. The llama.cpp flags now carry --cache-ram 0 and -ctxcp 4, and the fit counts slots x 4 x one checkpoint (the SWA cache of GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch), 256) cells plus the recurrent state) in the host bytes, the pod limit and every guard rung. On the CPU placement the checkpoints share the KV room, so the slot count accounts for them; on the GPU and experts-in-DRAM placements they are host bytes. A model with neither sliding-window nor linear-attention layers makes none. Tests: the Gemma QAT CPU plan on desktop-i9-nogpu emits --cache-ram 0 and -ctxcp K and counts seqs x K x 300 MiB in the host bytes, the pod limit and each rung's limit; Flash-Next's checkpoints are its recurrent state; a full-attention model makes none; k3d-gpu fit --json on the desktop shows the same.
On several GPUs the fit treated llama.cpp's cards as one pool: the summed budgets less the summed weights, KV and overhead. It then wrote -ts as each card's budget in GiB. llama.cpp places whole layers, though, and with --n-cpu-moe the layers whose experts stay on the GPU are the last ones. The Xeon fixture's Flash-Next IQ4_NL plan, -ts 30,30 --n-cpu-moe 28 at 262144 x 4, therefore put 43.5 GiB on the second 32 GiB card. The fit now sizes each card from the layers llama.cpp b11749 gives it with -ngl 999 (src/llama-model.cpp load_tensors): - layer il goes to the device upper_bound(cumsum(ts) / sum, il / (layers + 1)); - the output layer, holding lm_head, goes the same way and lands on the last card; - the first --n-cpu-moe layers keep their experts in host RAM; - each layer's KV or recurrent state sits on its card; - every card carries its own overhead. -ts is now integer layer counts summing to layers + 1, so each card gets exactly the layers it was sized for. Every card is held to the smallest budget of the cards used, because the device plugin orders them inside the pod. The largest card load is balanced, and a split that cannot fit is refused. The experts-in-DRAM placement uses the same per-card search, so -ts and --n-cpu-moe are chosen together. --explain lists each card, and --json has card_bytes. Flash-Next IQ4_NL on the Xeon is now -ts 35,14 --n-cpu-moe 29 at 262144 x 4: 29.9 and 28.9 GiB on cards with 30.4 and 30.2 GiB budgets. Tests: - llama_layer_devices matches the upper_bound rule. - The old -ts 30,30 --n-cpu-moe 28 plan works out to 43.5 GiB on its second card. - For the Flash-Next IQ4_NL plan and a two-card Gemma QAT layer split, each card's bytes, recomputed in the test from the flags, stay within budget and agree with card_bytes. - A split that cannot fit is refused. - fit_test.sh checks the Flash-Next plan's -ts and card_bytes.
…picks its card count
The per-layer weights counted the MTP head for every format, so a model sized from
safetensors for a GGUF conversion carried 4.9 GiB (Qwen3.8-Flash-Next) on its output
layer that model_profile leaves out of the bytes to serve. One-card moe-offload plans
therefore no longer matched the old arithmetic: Intel/Qwen3.8-Flash-Next-W4A16-AutoRound
with --gpus 1 on the fixture Xeon went from --n-cpu-moe 43 to 47. The output layer now
holds the bytes to serve less the lookup tables, experts, other and vision: lm_head, plus
MTP only when the GGUF holds it. The one-card plan is --n-cpu-moe 43 again.
The vision projector was put on the output layer, the last card. llama.cpp loads the
projector through tools/mtmd/clip.cpp, which takes the first GPU backend, so it now
counts with layer 0 on the first card, and --explain says so.
moe-offload always used every free card. Each card is held to the smallest budget of the
cards used, so a smaller card beside two large ones lowers the room on all of them. On
the fixture Xeon with 0, 0, 12 and 14 GiB in use on its cards (budgets 30.4, 30.4, 18.4
and 16.4 GiB), Intel/Qwen3.8-Flash-Next-W4A16-AutoRound at 262144 x 6 with --device gpu was refused
("do not fit 4 card(s) of 16.4 GiB each") while --gpus 2 fit. The fit now tries 1, 2,
... of the free cards by budget (or the --gpus count) and keeps the plan with the most
context, then the fewest CPU-expert layers, then the most sequences, then the fewest
cards; that case is now two cards, -ts 31,18 --n-cpu-moe 41. --gpus above the free card
count is refused, as the GPU placements already refuse it, instead of running on fewer
cards.
The fixture Xeon's GGUF plan is unchanged: -ts 35,14 --n-cpu-moe 29 at 262144 x 4.
Tests (each fails on the previous commit):
- llama_layer_bytes adds up to the bytes to serve less the lookup tables, its output
layer is lm_head alone and layer 0 carries the vision projector, for Qwen3.8-Flash-Next
from safetensors.
- On the mixed-budget Xeon, the automatic moe-offload plan fits, has no more CPU-expert
layers than any --gpus 1..4 plan, uses fewer than four cards and keeps every card in
budget.
- --gpus 3 on the fixture Xeon, two cards free, is refused for moe-offload.
- The test's own per-card arithmetic follows the same vision and MTP rule.
…checkpoints With #214, #216 and #217 together, the multi-card llama.cpp path (_llama_multi, which now also runs every moe-offload plan) had two gaps. Its per-layer KV counted a sliding-window layer at the window (1024 tokens for Gemma 4) while llama.cpp b11749 allocates GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch), 256) = 1536 cells per slot, so card_bytes disagreed with kv_bytes. And it left the context checkpoints out of the host bytes and the pod limit, which #216 counts on one card: the Flash-Next IQ4_NL plan on the Xeon fixture reported ckpt_bytes 0 with -ctxcp 4 in its flags. kv_parts now takes the sliding-window tokens from swa_tokens, so per_seq, the per-layer KV and the checkpoints share one cell count, and _llama_multi adds slots x checkpoints to the host bytes. The Flash-Next plan's pod limit goes from 74Gi to 76Gi. Tests: the Gemma split's per-card bytes are worked out at 1536 SWA cells and the KV over the cards equals kv_bytes; a split plan counts its checkpoints in host bytes; the Flash-Next two-card shell check asserts the checkpoints are on the host.
A model that ties its embeddings (Gemma 4) has no output.weight. llama.cpp b11749 then loads token_embd a second time as the output tensor (TENSOR_DUPLICATED), and src/llama-model-loader.cpp gives that copy the output layer's buffer type: the last card of a split, the one card otherwise. The input copy stays on the CPU, which the fit already counts as a host lookup table, but the output copy was counted nowhere, so the Gemma 4 QAT plans were 0.56 GiB short on the card holding the output layer. llama_output_copy names those bytes; llama_layer_bytes puts them on the output layer and the one-card llama.cpp placement adds them to its weights. --explain says so on both paths. On the CPU the copy maps the same file pages, so the CPU placement is unchanged. The fixture plans keep their context and sequence counts. Tests, failing before the change: the Gemma two-card split's per-card bytes with the copy on the output card, the one-card Gemma plan's GPU bytes with the copy, and a shell check that --explain names it.
llama.cpp b11749 assigns devices over n_layer_all, the GGUF's block count, which counts the MTP (nextn) blocks a GGUF keeps after its trunk (src/llama-model.cpp load_tensors, get_layer_buft_list). Intel/Qwen3.8-27B-q4km-AutoRound's GGUF holds one: qwen35.block_count 65 and qwen35.nextn_predict_layers 1 for a 64-layer trunk, with blk.64.nextn.* in its header. The per-card split counted only the trunk, so its -ts summed to 65 where llama.cpp spreads 66 slots, and the first card got one trunk layer more than it was sized for (-ts 34,31 put layers 0-34 there, not 0-33). model_profile now records the MTP blocks a GGUF holds (the config's mtp_num_hidden_layers or num_nextn_predict_layers, when the GGUF has MTP tensors), and llama_layer_bytes puts them after the trunk with their MTP tensors and no KV: the cache covers il < n_layer(), the trunk only. The output layer holds lm_head (or the tied embedding's copy) alone. The same plan is now -ts 34,32, and each card's bytes match llama.cpp's assignment over 66 slots. --explain names the MTP block. Plans of GGUFs without MTP tensors (Gemma 4, the Flash-Next GGUFs) are unchanged. Test written first and failing before the change: the Qwen3.8 27B GGUF split on two cards, its -ts summing to 66 and its per-card bytes worked out the way llama.cpp assigns them.
…nts them The agent converts safetensors that only llama.cpp runs here (Qwen3.8-Flash-Next) with convert_hf_to_gguf.py and no --no-mtp. llama.cpp v0.8.0's converter then keeps the MTP head after the trunk (conversion/qwen.py _QwenMtpMixin adds mtp_num_hidden_layers to block_count), so the GGUF it writes has 49 blocks for Flash-Next's 48 trunk layers, and llama-server assigns devices over all 49 plus the output layer. The fit sized the conversion as safetensors, without the MTP block, so its -ts summed to 49 and put one more trunk layer on the first card than it sized (-ts 39,10). llama_profile is the model llama.cpp serves: a GGUF as it is, other weights as the GGUF the conversion writes, with the MTP blocks and their bytes, the way a GGUF repo that holds an MTP head is already sized. vLLM and OVMS keep the profile without them. Flash-Next on the Xeon fixture is now -ts 39,11 --n-cpu-moe 45, every card within its budget, and --explain names the conversion's MTP block.
The fit now hands llama-server --cache-ram 0 and -ctxcp 4 (the host caches it counts) and -ts as layer counts per card. Both options are in llama.cpp v0.8.0's parser (common/arg.cpp: -ctxcp/--ctx-checkpoints :1709, -cram/--cache-ram :1727), so the args check takes them, and the rendered servers are checked for them. The QAT GGUF on one free B70 is 131072 x 5 (-c 655360 -np 5): the card also holds the tied embedding's output copy. Flash-Next on two B70s is -ts 39,11 --n-cpu-moe 45 -np 4, the split sized card by card with the conversion's MTP block.
The OVMS fix passed --gpus to the fit only when the agent app was vLLM, so an OVMS pod, which holds one gpu.intel.com/xe, is no longer sized for two cards. The llama.cpp backend splits a model's layers over several cards and takes its card count from the fit, so with that rule K3D_GPU_AGENT_BACKEND=llama.cpp K3D_GPU_AGENT_GPUS=2 lost the 2 and served the QAT GGUF on one card. --gpus now reaches the fit for every app but OVMS: the QAT GGUF is split over the two free B70s with -ts in layers, and the OVMS pod stays on one card. The help and docs/fit.md say which backends read it.
The planner fixes read llama.cpp b11749; the agent's llama-server is now v0.8.0 (b11960). Every rule they size by holds there unchanged: the SWA cache of GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch), 256) cells (src/llama-kv-cache-iswa.cpp:73, n_ctx_seq padded to 256 in src/llama-context.cpp:393), at most -ctxcp checkpoints per slot made at the last user message and 4 + n_ubatch and 4 tokens before the prompt's end (common/common.h:641, tools/server/server-context.cpp:2746 and :4502-4518), the 8192 MiB --cache-ram default (common/common.h:644), the upper_bound layer split over n_layer_all + 1 with the output layer last (src/llama-model.cpp:1685-1710), and the tied embedding's output copy (src/llama-model-loader.cpp:1176, :1381). The comments, the tests' comments and docs/fit.md cite v0.8.0, and _flags cites the parser the flags were checked against.
…nd OVMS does not serve the model With K3D_GPU_AGENT_BACKEND unset, AGENT_APP is OVMS's until the fit picks the engine, so agent_fit dropped --gpus for every model the optimizer did not first move to another backend, not just the ones OVMS serves. A model the fit then sent to vLLM or llama.cpp lost the cards asked for: K3D_GPU_AGENT_GPUS=2 with K3D_GPU_AGENT_MODEL set to the QAT GGUF, or with K3D_GPU_MODEL_OPT=off and RedHatAI/Qwen3.8-27B-INT4, rendered one card where main renders two. With the backend set, --gpus is still left out for OVMS alone. With it unset, it is left out for an OpenVINO IR, the only weights the fit gives to OVMS (FORMAT_ENGINE); any other model gets it. The OVMS test from the one-GPU fix still renders one card. New tests (each failed before the change): the INT4 checkpoint with the backend unset on two cards in tests/fit_test.sh, and the QAT GGUF from K3D_GPU_AGENT_MODEL with the backend unset split over two B70s in tests/llamacpp_test.sh.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands #229 through the batch train, with one CI run. It is merged with --no-ff, so its commits keep their own history.
#229 supersedes #212, #214, #216 and #217; they are closed once this lands.
No ledger rows change, so docs/intel/technical-report.md and console/ledger-rules.json need no update. Local checks on the batch head: every suite in .github/workflows/ci.yml that runs without extra tooling, render_test with the pinned helm, the simulated hardware matrix, the agent examples, tests/claims_check.sh and tests/ledger_drift_check.py.