[88plug]

Problem

On several GPUs the fit sized llama.cpp's cards as one pool: the summed budgets against the summed weights, KV and overhead. It then wrote -ts as each card's budget in GiB.

llama.cpp places whole layers. With --n-cpu-moe, the layers that keep their experts on the GPU are the last ones. So the Xeon fixture's Flash-Next IQ4_NL plan, -ts 30,30 --n-cpu-moe 28 at 262144 x 4, put 43.5 GiB on the second 32 GiB card.

How llama.cpp b11749 assigns layers

src/llama-model.cpp load_tensors, with -ngl 999:

• layer il goes to device upper_bound(cumsum(ts)/sum, il/(n_layer+1));
• the output layer is il = n_layer, which is the last card;
• --n-cpu-moe N adds CPU buffer overrides for blk.<i>.ffn_*_exps with i < N (common/common.h llm_add_n_cpu_ffn_overrides).

Changes

console/fit.py:

• llama_layer_devices(ts, n_layer) follows the rule above exactly.
• llama_layer_bytes gives each layer's GPU weights and KV:
  • weights are the class totals spread over the layers, minus the experts of the first --n-cpu-moe layers;
  • KV comes from config.json layer_types, and each layer's KV or recurrent state sits on its card;
  • the output layer holds lm_head (and the MTP and vision tensors, when the GGUF has them).
• llama_split picks integer layer counts summing to n_layer + 1, so every card gets exactly the layers it was sized for. It keeps every card's layers plus its own overhead within the smallest budget of the cards used, because the device plugin orders the cards inside the pod. It balances the largest card's load, and returns nothing when no split fits.
• The layer-split placement and the experts-in-DRAM placement share one search (_llama_multi): the most context first, then the fewest CPU-expert layers, then the most sequences. On one card that search gives the same arithmetic as before.
• --explain lists each card's layers, weights, KV and total. --json gains card_bytes, and gpu_bytes is the largest card.

docs/fit.md: the per-card rule and -ts as layer counts.

Result on the Xeon fixture: Flash-Next IQ4_NL becomes -ts 35,14 --n-cpu-moe 29 at 262144 x 4, with 29.9 and 28.9 GiB on cards whose budgets are 30.4 and 30.2 GiB.

Tests

tests/fit_test.py LlamaSplit:

• llama_layer_devices matches upper_bound, including a 0 entry;
• the old -ts 30,30 --n-cpu-moe 28 plan, worked out independently in the test, puts more than 32 GiB on its second card;
• for the Flash-Next IQ4_NL plan and a two-card Gemma QAT layer split (262144 x 6), every card's bytes are recomputed in the test from the emitted flags, stay within the budget, and agree with card_bytes;
• a split that cannot fit is refused.

tests/fit_test.sh: the Flash-Next IQ4_NL --json plan has integer -ts summing to 49, is not 30,30, and every card_bytes entry is within budget.

The new tests fail on origin/main's fit.py.

An extra sweep, not committed, covered every fixture model and host x --gpus None/1/2/3 x --device None/gpu x three ctx/seqs settings: 1152 plans, no exceptions, and 92 llama.cpp split plans each checked per card against budget, with no violation.

Run before opening:

• tests/fit_test.sh: 35 passed
• python3 tests/fit_test.py: 61 tests OK
• tests/model_opt_test.sh: 37 passed
• tests/claims_check.sh, python3 tests/ledger_drift_check.py, ruff (F, E9), shellcheck v0.11.0 at warning severity: clean

Not changed, found while here

A GGUF with tied embeddings has no output.weight. Gemma 4 is one: in src/models/gemma4.cpp, output falls back to token_embd with TENSOR_DUPLICATED. llama.cpp then keeps a copy of token_embd on the output layer's device, 0.56 GiB for the Gemma QAT fixture. No placement counts that copy today, single-card included. It is left for a follow-up because it changes single-GPU plans as well.

Overlap

#214 and #216 also touch per_seq and the llama.cpp flags in console/fit.py. Whichever lands later needs a rebase. Here per_seq goes through kv_parts, so #214's swa_tokens belongs inside kv_parts.

[88plug]

Superseded by #229 (merged)