Repository navigation
fit: the four planner fixes together, and where they meet - #229
Merged
Merged
Conversation
88plug
force-pushed
the
claude/fit-plans-fix
branch
from
October 29, 2026 09:20
2c778b6 to
6e1ca17
Compare
88plug
changed the title
With K3D_GPU_AGENT_GPUS=2 the launcher passed --gpus 2 to the fit for an OVMS agent too. The fit then sized a layer split over two cards and rendered a KV pool of two cards' worth (--cache_size=41 for the Gemma 4 int4 IR on the Xeon fixture) into a pod that holds one gpu.intel.com/xe. The fit now refuses a multi-GPU OVMS placement with "OVMS serves one GPU", and agent_fit passes --gpus only when the agent app is vLLM. When no placement can be tried at all, the refusal says no Intel GPU with device-local memory is free instead of echoing the engine note. Tests: OVMS with --gpus 2 is refused, the GPU cache_size stays within one card's budget, the no-free-GPU reason, and an agent manifest with K3D_GPU_AGENT_GPUS=2 under OVMS whose --cache_size fits one card.
The fit capped every engine's sliding-window layers at the window. OpenVINO GenAI 2026.4 has no sliding-window cache type (only KV_CACHE and LINEAR_ATTENTION_CACHE in continuous_batching/cache/cache_type.hpp), and one KV block spans every attention layer, so OVMS holds every token in those layers too. Gemma 4 costs 225280 B per token under OVMS, not 20480 B plus a 1024-token window; on the Xeon fixture the old fit set --cache_size=13 for 131072 x 5 sequences, which needs about 138 GiB. kv_profile and per_seq are now engine-aware through swa_tokens: OVMS counts sliding-window layers at the whole context, vLLM keeps the window cap, and llama.cpp counts its SWA cache of GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch 512), 256) cells per slot (src/llama-kv-cache-iswa.cpp at b11749), 1536 cells for Gemma 4's 1024 window. OVMS --cache_size is whole GiB and was floored, so the pool could hold fewer tokens than the fit counted. Whole-GiB pools (OVMS, and vLLM's CPU KV space) are now sized inside whole GiB of room and rounded up, on every guard rung too. Tests: per-engine SWA bytes for Gemma 4; the Gemma 4 OVMS cache_size holds context x sequences at 225280 B per token on a GPU, on the Xeon CPU and on the desktop, at every guard rung; the agent manifest renders such a pool; the Gemma QAT GGUF plan drops from 6 to 5 slots for the 1536-cell SWA cache.
llama-server b11749 keeps two host caches beside the KV cache that the fit did not count: a prompt cache of up to 8192 MiB (--cache-ram) and up to 32 context checkpoints per slot (-ctxcp), each a host copy of the slot's SWA cache and recurrent state. For the Gemma QAT GGUF on the desktop CPU plan (5 slots at 262144, KV filling the room) that is 8 GiB plus about 47 GiB of checkpoints over a 45Gi pod limit. The llama.cpp flags now carry --cache-ram 0 and -ctxcp 4, and the fit counts slots x 4 x one checkpoint (the SWA cache of GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch), 256) cells plus the recurrent state) in the host bytes, the pod limit and every guard rung. On the CPU placement the checkpoints share the KV room, so the slot count accounts for them; on the GPU and experts-in-DRAM placements they are host bytes. A model with neither sliding-window nor linear-attention layers makes none. Tests: the Gemma QAT CPU plan on desktop-i9-nogpu emits --cache-ram 0 and -ctxcp K and counts seqs x K x 300 MiB in the host bytes, the pod limit and each rung's limit; Flash-Next's checkpoints are its recurrent state; a full-attention model makes none; k3d-gpu fit --json on the desktop shows the same.
On several GPUs the fit treated llama.cpp's cards as one pool: the summed budgets less the summed weights, KV and overhead. It then wrote -ts as each card's budget in GiB. llama.cpp places whole layers, though, and with --n-cpu-moe the layers whose experts stay on the GPU are the last ones. The Xeon fixture's Flash-Next IQ4_NL plan, -ts 30,30 --n-cpu-moe 28 at 262144 x 4, therefore put 43.5 GiB on the second 32 GiB card. The fit now sizes each card from the layers llama.cpp b11749 gives it with -ngl 999 (src/llama-model.cpp load_tensors): - layer il goes to the device upper_bound(cumsum(ts) / sum, il / (layers + 1)); - the output layer, holding lm_head, goes the same way and lands on the last card; - the first --n-cpu-moe layers keep their experts in host RAM; - each layer's KV or recurrent state sits on its card; - every card carries its own overhead. -ts is now integer layer counts summing to layers + 1, so each card gets exactly the layers it was sized for. Every card is held to the smallest budget of the cards used, because the device plugin orders them inside the pod. The largest card load is balanced, and a split that cannot fit is refused. The experts-in-DRAM placement uses the same per-card search, so -ts and --n-cpu-moe are chosen together. --explain lists each card, and --json has card_bytes. Flash-Next IQ4_NL on the Xeon is now -ts 35,14 --n-cpu-moe 29 at 262144 x 4: 29.9 and 28.9 GiB on cards with 30.4 and 30.2 GiB budgets. Tests: - llama_layer_devices matches the upper_bound rule. - The old -ts 30,30 --n-cpu-moe 28 plan works out to 43.5 GiB on its second card. - For the Flash-Next IQ4_NL plan and a two-card Gemma QAT layer split, each card's bytes, recomputed in the test from the flags, stay within budget and agree with card_bytes. - A split that cannot fit is refused. - fit_test.sh checks the Flash-Next plan's -ts and card_bytes.
…picks its card count
The per-layer weights counted the MTP head for every format, so a model sized from
safetensors for a GGUF conversion carried 4.9 GiB (Qwen3.8-Flash-Next) on its output
layer that model_profile leaves out of the bytes to serve. One-card moe-offload plans
therefore no longer matched the old arithmetic: Intel/Qwen3.8-Flash-Next-W4A16-AutoRound
with --gpus 1 on the fixture Xeon went from --n-cpu-moe 43 to 47. The output layer now
holds the bytes to serve less the lookup tables, experts, other and vision: lm_head, plus
MTP only when the GGUF holds it. The one-card plan is --n-cpu-moe 43 again.
The vision projector was put on the output layer, the last card. llama.cpp loads the
projector through tools/mtmd/clip.cpp, which takes the first GPU backend, so it now
counts with layer 0 on the first card, and --explain says so.
moe-offload always used every free card. Each card is held to the smallest budget of the
cards used, so a smaller card beside two large ones lowers the room on all of them. On
the fixture Xeon with 0, 0, 12 and 14 GiB in use on its cards (budgets 30.4, 30.4, 18.4
and 16.4 GiB), Intel/Qwen3.8-Flash-Next-W4A16-AutoRound at 262144 x 6 with --device gpu was refused
("do not fit 4 card(s) of 16.4 GiB each") while --gpus 2 fit. The fit now tries 1, 2,
... of the free cards by budget (or the --gpus count) and keeps the plan with the most
context, then the fewest CPU-expert layers, then the most sequences, then the fewest
cards; that case is now two cards, -ts 31,18 --n-cpu-moe 41. --gpus above the free card
count is refused, as the GPU placements already refuse it, instead of running on fewer
cards.
The fixture Xeon's GGUF plan is unchanged: -ts 35,14 --n-cpu-moe 29 at 262144 x 4.
Tests (each fails on the previous commit):
- llama_layer_bytes adds up to the bytes to serve less the lookup tables, its output
layer is lm_head alone and layer 0 carries the vision projector, for Qwen3.8-Flash-Next
from safetensors.
- On the mixed-budget Xeon, the automatic moe-offload plan fits, has no more CPU-expert
layers than any --gpus 1..4 plan, uses fewer than four cards and keeps every card in
budget.
- --gpus 3 on the fixture Xeon, two cards free, is refused for moe-offload.
- The test's own per-card arithmetic follows the same vision and MTP rule.
…checkpoints With #214, #216 and #217 together, the multi-card llama.cpp path (_llama_multi, which now also runs every moe-offload plan) had two gaps. Its per-layer KV counted a sliding-window layer at the window (1024 tokens for Gemma 4) while llama.cpp b11749 allocates GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch), 256) = 1536 cells per slot, so card_bytes disagreed with kv_bytes. And it left the context checkpoints out of the host bytes and the pod limit, which #216 counts on one card: the Flash-Next IQ4_NL plan on the Xeon fixture reported ckpt_bytes 0 with -ctxcp 4 in its flags. kv_parts now takes the sliding-window tokens from swa_tokens, so per_seq, the per-layer KV and the checkpoints share one cell count, and _llama_multi adds slots x checkpoints to the host bytes. The Flash-Next plan's pod limit goes from 74Gi to 76Gi. Tests: the Gemma split's per-card bytes are worked out at 1536 SWA cells and the KV over the cards equals kv_bytes; a split plan counts its checkpoints in host bytes; the Flash-Next two-card shell check asserts the checkpoints are on the host.
A model that ties its embeddings (Gemma 4) has no output.weight. llama.cpp b11749 then loads token_embd a second time as the output tensor (TENSOR_DUPLICATED), and src/llama-model-loader.cpp gives that copy the output layer's buffer type: the last card of a split, the one card otherwise. The input copy stays on the CPU, which the fit already counts as a host lookup table, but the output copy was counted nowhere, so the Gemma 4 QAT plans were 0.56 GiB short on the card holding the output layer. llama_output_copy names those bytes; llama_layer_bytes puts them on the output layer and the one-card llama.cpp placement adds them to its weights. --explain says so on both paths. On the CPU the copy maps the same file pages, so the CPU placement is unchanged. The fixture plans keep their context and sequence counts. Tests, failing before the change: the Gemma two-card split's per-card bytes with the copy on the output card, the one-card Gemma plan's GPU bytes with the copy, and a shell check that --explain names it.
llama.cpp b11749 assigns devices over n_layer_all, the GGUF's block count, which counts the MTP (nextn) blocks a GGUF keeps after its trunk (src/llama-model.cpp load_tensors, get_layer_buft_list). Intel/Qwen3.8-27B-q4km-AutoRound's GGUF holds one: qwen35.block_count 65 and qwen35.nextn_predict_layers 1 for a 64-layer trunk, with blk.64.nextn.* in its header. The per-card split counted only the trunk, so its -ts summed to 65 where llama.cpp spreads 66 slots, and the first card got one trunk layer more than it was sized for (-ts 34,31 put layers 0-34 there, not 0-33). model_profile now records the MTP blocks a GGUF holds (the config's mtp_num_hidden_layers or num_nextn_predict_layers, when the GGUF has MTP tensors), and llama_layer_bytes puts them after the trunk with their MTP tensors and no KV: the cache covers il < n_layer(), the trunk only. The output layer holds lm_head (or the tied embedding's copy) alone. The same plan is now -ts 34,32, and each card's bytes match llama.cpp's assignment over 66 slots. --explain names the MTP block. Plans of GGUFs without MTP tensors (Gemma 4, the Flash-Next GGUFs) are unchanged. Test written first and failing before the change: the Qwen3.8 27B GGUF split on two cards, its -ts summing to 66 and its per-card bytes worked out the way llama.cpp assigns them.
…nts them The agent converts safetensors that only llama.cpp runs here (Qwen3.8-Flash-Next) with convert_hf_to_gguf.py and no --no-mtp. llama.cpp v0.8.0's converter then keeps the MTP head after the trunk (conversion/qwen.py _QwenMtpMixin adds mtp_num_hidden_layers to block_count), so the GGUF it writes has 49 blocks for Flash-Next's 48 trunk layers, and llama-server assigns devices over all 49 plus the output layer. The fit sized the conversion as safetensors, without the MTP block, so its -ts summed to 49 and put one more trunk layer on the first card than it sized (-ts 39,10). llama_profile is the model llama.cpp serves: a GGUF as it is, other weights as the GGUF the conversion writes, with the MTP blocks and their bytes, the way a GGUF repo that holds an MTP head is already sized. vLLM and OVMS keep the profile without them. Flash-Next on the Xeon fixture is now -ts 39,11 --n-cpu-moe 45, every card within its budget, and --explain names the conversion's MTP block.
The fit now hands llama-server --cache-ram 0 and -ctxcp 4 (the host caches it counts) and -ts as layer counts per card. Both options are in llama.cpp v0.8.0's parser (common/arg.cpp: -ctxcp/--ctx-checkpoints :1709, -cram/--cache-ram :1727), so the args check takes them, and the rendered servers are checked for them. The QAT GGUF on one free B70 is 131072 x 5 (-c 655360 -np 5): the card also holds the tied embedding's output copy. Flash-Next on two B70s is -ts 39,11 --n-cpu-moe 45 -np 4, the split sized card by card with the conversion's MTP block.
The OVMS fix passed --gpus to the fit only when the agent app was vLLM, so an OVMS pod, which holds one gpu.intel.com/xe, is no longer sized for two cards. The llama.cpp backend splits a model's layers over several cards and takes its card count from the fit, so with that rule K3D_GPU_AGENT_BACKEND=llama.cpp K3D_GPU_AGENT_GPUS=2 lost the 2 and served the QAT GGUF on one card. --gpus now reaches the fit for every app but OVMS: the QAT GGUF is split over the two free B70s with -ts in layers, and the OVMS pod stays on one card. The help and docs/fit.md say which backends read it.
The planner fixes read llama.cpp b11749; the agent's llama-server is now v0.8.0 (b11960). Every rule they size by holds there unchanged: the SWA cache of GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch), 256) cells (src/llama-kv-cache-iswa.cpp:73, n_ctx_seq padded to 256 in src/llama-context.cpp:393), at most -ctxcp checkpoints per slot made at the last user message and 4 + n_ubatch and 4 tokens before the prompt's end (common/common.h:641, tools/server/server-context.cpp:2746 and :4502-4518), the 8192 MiB --cache-ram default (common/common.h:644), the upper_bound layer split over n_layer_all + 1 with the output layer last (src/llama-model.cpp:1685-1710), and the tied embedding's output copy (src/llama-model-loader.cpp:1176, :1381). The comments, the tests' comments and docs/fit.md cite v0.8.0, and _flags cites the parser the flags were checked against.
88plug
force-pushed
the
claude/fit-plans-fix
branch
from
October 29, 2026 20:28
f64cdb9 to
46d55b3
Compare
…nd OVMS does not serve the model With K3D_GPU_AGENT_BACKEND unset, AGENT_APP is OVMS's until the fit picks the engine, so agent_fit dropped --gpus for every model the optimizer did not first move to another backend, not just the ones OVMS serves. A model the fit then sent to vLLM or llama.cpp lost the cards asked for: K3D_GPU_AGENT_GPUS=2 with K3D_GPU_AGENT_MODEL set to the QAT GGUF, or with K3D_GPU_MODEL_OPT=off and RedHatAI/Qwen3.8-27B-INT4, rendered one card where main renders two. With the backend set, --gpus is still left out for OVMS alone. With it unset, it is left out for an OpenVINO IR, the only weights the fit gives to OVMS (FORMAT_ENGINE); any other model gets it. The OVMS test from the one-GPU fix still renders one card. New tests (each failed before the change): the INT4 checkpoint with the backend unset on two cards in tests/fit_test.sh, and the QAT GGUF from K3D_GPU_AGENT_MODEL with the backend unset split over two B70s in tests/llamacpp_test.sh.
Owner
Author
|
Review: one defect found and fixed in e49d0f8. The OVMS one-GPU gate in agent_fit keyed on AGENT_APP, which is OVMS's whenever K3D_GPU_AGENT_BACKEND is unset (the default) until the fit picks the engine. So K3D_GPU_AGENT_GPUS=2 was dropped for vLLM and llama.cpp models too: K3D_GPU_AGENT_MODEL=google/gemma-4-26B-A4B-it-qat-q4_0-gguf rendered one card (main: -ts over two), and K3D_GPU_MODEL_OPT=off with RedHatAI/Qwen3.8-27B-INT4 rendered tensor parallel 1 (main: 2). With the backend unset the gate now leaves --gpus out for an OpenVINO IR alone, the only weights fit.py's FORMAT_ENGINE gives to OVMS; with it set, for OVMS alone as before. Two new tests, each failing before the change. Rest of the branch re-derived against llama.cpp v0.8.0 sources (kv-cache-iswa size_swa, layer split upper_bound over n_layer_all + 1, TENSOR_DUPLICATED output copy, -ctxcp/--cache-ram defaults and parser, qwen4exp converter keeping the MTP block through _QwenMtpMixin): the cited rules hold. Run after the fix: fit_test.sh 40 passed (fit_test.py 83 OK), llamacpp_test.sh 49, detect_test.sh 1185, claims_check 26 files clean, ledger_drift_check 41 files agree with 81 rows, claims and drift self-tests pass, shellcheck v0.11.0 warning-clean on the three changed shell files. Merges cleanly with main at 74ff63e. |
Merged
88plug
added a commit
that referenced
this pull request
Oct 30, 2026
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four fit planner fixes, #212, #214, #216 and #217, in one branch with the places where they meet resolved, rebased onto main after #182 (the llama.cpp agent backend and busy-card pinning) and integrated with it. The four touch the same functions in console/fit.py (per_seq, the moe-offload placement, the fit/v1 fields, the llama.cpp flags) and the same docs/fit.md table, so they conflict with each other once any one lands. Land this in place of the four; #212, #214, #216 and #217 are closed, pointing here, once it lands.
The branch is linear on main: the four fixes as their own commits (#217's two), the three fixes where they meet, then four commits that fit them to #182, and one that keeps K3D_GPU_AGENT_GPUS with the backend unset.
What each brings (details in each PR):
fit --gpus 2with OVMS is refused as "OVMS serves one GPU", and agent_fit does not pass--gpusto an OVMS agent. With K3D_GPU_AGENT_GPUS=2 an OVMS pod used to get--cache_size=41beside onegpu.intel.com/xe. A fit that can try no placement at all now says no Intel GPU with device-local memory is free.--cache_sizeis sized and rounded up in whole GiB to hold context x sequences. llama.cpp gives a slot GGML_PAD(min(n_ctx_seq, n_swa + 512), 256) cells (src/llama-kv-cache-iswa.cpp). vLLM keeps the window.--cache-ram 0 -ctxcp 4, and slots x 4 x one checkpoint (the slot's SWA cache and recurrent state) count in the host bytes and the pod limit.--n-cpu-moe Nexperts on the CPU; the output layer on the last card).-tsis emitted as whole layer counts that keep every card within budget, or the placement is refused. The Xeon fixture's Flash-Next IQ4_NL plan goes from-ts 30,30 --n-cpu-moe 28, which put 43.5 GiB on the second 32 GiB card, to-ts 35,14 --n-cpu-moe 29at 29.9 and 28.9 GiB.Fixes where the four meet (tests for each were written first and failed before the change):
--explainnames it.-tssums to 66 (-ts 34,32, not34,31, which put one more trunk layer on the first card than it was sized for).Integration with #182:
K3D_GPU_AGENT_GPUS reaches the fit for llama.cpp as for vLLM. fit: OVMS serves one GPU #212's launcher change passed
--gpusonly to a vLLM agent; agent: llama.cpp backend the fit picks, and busy Intel cards left out of the device plugin #182's llama.cpp backend splits layers over several cards and takes its card count from the fit, soK3D_GPU_AGENT_BACKEND=llama.cpp K3D_GPU_AGENT_GPUS=2lost the 2 and served the QAT GGUF on one card.--gpusnow reaches the fit for every app but OVMS. New test in tests/llamacpp_test.sh (failed before the change): the QAT GGUF split over the two free B70s,-tsin layers, twogpu.intel.com/xe. The OVMS test from fit: OVMS serves one GPU #212 still renders one card.A GGUF conversion keeps the MTP blocks. agent: llama.cpp backend the fit picks, and busy Intel cards left out of the device plugin #182 converts Qwen3.8-Flash-Next with convert_hf_to_gguf.py and no
--no-mtp, and llama.cpp v0.8.0's converter keeps the MTP head (conversion/qwen.py_QwenMtpMixinadds mtp_num_hidden_layers to block_count): 49 blocks for 48 trunk layers. fit: size llama.cpp's layer split card by card #217's-tssummed to 49 (-ts 39,10), one trunk layer more on the first card than it sized.llama_profileis the model llama.cpp serves: a GGUF as it is, other weights as the GGUF the conversion writes, with the MTP blocks and their bytes, as a GGUF repo that holds an MTP head is sized. vLLM and OVMS keep the profile without them. Flash-Next is now-ts 39,11 --n-cpu-moe 45 -np 4, every card within budget. Tests in tests/fit_test.py (failed before the change): the conversion's split checked card by card against the converted GGUF's 50 slots, and the layer bytes with the MTP block.The fit's flags reach llama-server:
--cache-ram 0,-ctxcp 4and-tsas layer counts go through agent: llama.cpp backend the fit picks, and busy Intel cards left out of the device plugin #182's knobs into the llamacpp Deployment. agent: llama.cpp backend the fit picks, and busy Intel cards left out of the device plugin #182's args check takes-ctxcpand--cache-ram, both in llama.cpp v0.8.0's parser (common/arg.cpp :1709 and :1727), and the rendered servers are checked for them. The QAT GGUF on one free B70 is now 131072 x 5 (-c 655360 -np 5): the card also holds the tied embedding's output copy.llama.cpp v0.8.0 cited: the four fixes read llama.cpp b11749, the agent runs v0.8.0 (b11960). Every rule they size by holds there unchanged (src/llama-kv-cache-iswa.cpp:73, src/llama-context.cpp:393, common/common.h:641 and :644, tools/server/server-context.cpp:2746 and :4502-4518, src/llama-model.cpp:1685-1710, src/llama-model-loader.cpp:1176 and :1381), and the comments and docs/fit.md cite v0.8.0.
K3D_GPU_AGENT_GPUS reaches the fit with the backend unset. AGENT_APP is OVMS's until the fit picks the engine, so the gate above dropped
--gpusfor every model the optimizer did not first move to another backend:K3D_GPU_AGENT_GPUS=2withK3D_GPU_AGENT_MODELset to the QAT GGUF, or withK3D_GPU_MODEL_OPT=offand RedHatAI/Qwen3.8-27B-INT4, rendered one card where main renders two. With the backend unset,--gpusis now left out for an OpenVINO IR alone, the only weights the fit gives to OVMS. New tests in tests/fit_test.sh and tests/llamacpp_test.sh (each failed before the change).#182's busy cards and #212's refusal agree: a card others hold at half its VRAM or more is out of the fit's budget (card_busy) and out of the device plugin's view, and a fit with no free card left says no Intel GPU with device-local memory is free.
Rebase conflicts: console/fit.py (the fit/v1 document carries #182's source, files and convert beside ckpt_bytes and card_bytes), docs/fit.md (#182's backend sections kept; the flags table row merged:
-ts(layers per card),--cache-ram 0,-ctxcp 4,-ctk/-ctv), scripts/k3d-gpu (#182's loop that stops every running backend kept around the--gpusline).Seen while integrating, not changed here:
--spec-type draft-mtp(src/llama-model.cpp nextn_flags, TENSOR_SKIP otherwise). The fit counts them either way, as main already does for a GGUF repo: an upper bound when drafting is off.Run on the branch before pushing (fit, llamacpp, claims and drift again after the last rebase onto main at d997ea8, which touched no fit, launcher or docs/fit.md path):