Repository navigation
Conversation
The fit capped every engine's sliding-window layers at the window. OpenVINO GenAI 2026.4 has no sliding-window cache type (only KV_CACHE and LINEAR_ATTENTION_CACHE in continuous_batching/cache/cache_type.hpp), and one KV block spans every attention layer, so OVMS holds every token in those layers too. Gemma 4 costs 225280 B per token under OVMS, not 20480 B plus a 1024-token window; on the Xeon fixture the old fit set --cache_size=13 for 131072 x 5 sequences, which needs about 138 GiB. kv_profile and per_seq are now engine-aware through swa_tokens: OVMS counts sliding-window layers at the whole context, vLLM keeps the window cap, and llama.cpp counts its SWA cache of GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch 512), 256) cells per slot (src/llama-kv-cache-iswa.cpp at b11749), 1536 cells for Gemma 4's 1024 window. OVMS --cache_size is whole GiB and was floored, so the pool could hold fewer tokens than the fit counted. Whole-GiB pools (OVMS, and vLLM's CPU KV space) are now sized inside whole GiB of room and rounded up, on every guard rung too. Tests: per-engine SWA bytes for Gemma 4; the Gemma 4 OVMS cache_size holds context x sequences at 225280 B per token on a GPU, on the Xeon CPU and on the desktop, at every guard rung; the agent manifest renders such a pool; the Gemma QAT GGUF plan drops from 6 to 5 slots for the 1536-cell SWA cache.
This was referenced Oct 29, 2026
88plug
added a commit
that referenced
this pull request
Oct 29, 2026
88plug
added a commit
that referenced
this pull request
Oct 29, 2026
…checkpoints With #214, #216 and #217 together, the multi-card llama.cpp path (_llama_multi, which now also runs every moe-offload plan) had two gaps. Its per-layer KV counted a sliding-window layer at the window (1024 tokens for Gemma 4) while llama.cpp b11749 allocates GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch), 256) = 1536 cells per slot, so card_bytes disagreed with kv_bytes. And it left the context checkpoints out of the host bytes and the pod limit, which #216 counts on one card: the Flash-Next IQ4_NL plan on the Xeon fixture reported ckpt_bytes 0 with -ctxcp 4 in its flags. kv_parts now takes the sliding-window tokens from swa_tokens, so per_seq, the per-layer KV and the checkpoints share one cell count, and _llama_multi adds slots x checkpoints to the host bytes. The Flash-Next plan's pod limit goes from 74Gi to 76Gi. Tests: the Gemma split's per-card bytes are worked out at 1536 SWA cells and the KV over the cards equals kv_bytes; a split plan counts its checkpoints in host bytes; the Flash-Next two-card shell check asserts the checkpoints are on the host.
88plug
added a commit
that referenced
this pull request
Oct 29, 2026
…checkpoints With #214, #216 and #217 together, the multi-card llama.cpp path (_llama_multi, which now also runs every moe-offload plan) had two gaps. Its per-layer KV counted a sliding-window layer at the window (1024 tokens for Gemma 4) while llama.cpp b11749 allocates GGML_PAD(min(n_ctx_seq, n_swa + n_ubatch), 256) = 1536 cells per slot, so card_bytes disagreed with kv_bytes. And it left the context checkpoints out of the host bytes and the pod limit, which #216 counts on one card: the Flash-Next IQ4_NL plan on the Xeon fixture reported ckpt_bytes 0 with -ctxcp 4 in its flags. kv_parts now takes the sliding-window tokens from swa_tokens, so per_seq, the per-layer KV and the checkpoints share one cell count, and _llama_multi adds slots x checkpoints to the host bytes. The Flash-Next plan's pod limit goes from 74Gi to 76Gi. Tests: the Gemma split's per-card bytes are worked out at 1536 SWA cells and the KV over the cards equals kv_bytes; a split plan counts its checkpoints in host bytes; the Flash-Next two-card shell check asserts the checkpoints are on the host.
Merged
Owner
Author
|
Superseded by #229 (merged) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fit capped every engine's sliding-window layers at the model's window. That holds for vLLM only.
src/cpp/src/continuous_batching/cache/cache_type.hppat 2026.4.1.0 lists onlyKV_CACHEandLINEAR_ATTENTION_CACHE, andkv_cache_manager.hppsums one KV block's bytes over every decoder layer. Every attention layer therefore holds the whole context. Gemma 4 costs 225280 B per token under OVMS (5 full and 25 sliding layers, K and V at 2 B). On the Xeon fixture the old fit set--cache_size=13for 131072 x 5 sequences, which needs about 138 GiB.src/llama-kv-cache-iswa.cppsizes the SWA cache atGGML_PAD(min(size_base, n_swa*(unified ? n_seq_max : 1) + n_ubatch), 256). With-npset the cache is not unified, so each slot gets 1536 cells for Gemma 4's 1024 window at the default ubatch of 512.Changes:
kv_profile(cfg, kv_bytes, engine)records the engine's SWA cache mode, andper_seqtakes the SWA tokens fromswa_tokens: the whole context for OVMS, the window for vLLM, and the padded per-slot cache for llama.cpp. The explain line says which one applies and why.--cache_sizeis whole GiB and was floored, so the pool could hold fewer tokens than the fit counted. Whole-GiB pools (OVMS, and vLLM's CPUVLLM_CPU_KVCACHE_SPACE, which already rounded up) are now sized inside whole GiB of room and rounded up, on every guard rung as well.Effect on the fixtures: Gemma 4 int4 IR on one B70 goes from 131072 x 5 with a 13 GiB pool to 16384 x 4 with a 14 GiB pool, flagged below target. The Gemma QAT GGUF on one B70 goes from 6 to 5 slots at 131072.
Tests:
test_swa_cache_per_engine). The Gemma 4 OVMS--cache_sizeholds context x sequences at 225280 B per token on a Xeon GPU, the Xeon CPU and the desktop, and on every guard rung (test_gemma_ovms_cache_holds_every_token). The Gemma QAT GGUF slot count is updated to 5.--cache_sizethat holds its annotated seqs x context at 225280 B per token.Run before opening: tests/fit_test.sh (35 passed), python3 tests/fit_test.py (58 tests OK), tests/model_opt_test.sh (37 passed), tests/claims_check.sh, python3 tests/ledger_drift_check.py, ruff (F, E9), shellcheck v0.11.0 at warning severity.