Skip to content

ledger(H7x): amendment V1, arms B and C on EAT release-1.1.0 (vLLM CPU v0.28.0) #235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
88plug wants to merge 1 commit into from
Closed

ledger(H7x): amendment V1, arms B and C on EAT release-1.1.0 (vLLM CPU v0.28.0) #235

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 68 additions & 1 deletion EXPERIMENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim.
| H34x | H34's hypothesis on the Xeon 6741P: same-engine CPU Fast Draft (OVMS 2026.4.1, `OpenVINO/Qwen3-0.6B-int4-ov` draft on the CPU) beats the `OpenVINO/Qwen3-4B-int4-ov` target alone on single-stream decode, under matched background load. A replication: the original models are kept. | Written 2026-10-20, before any H34x measurement. At `typical`: median paired B/A decode tok/s 1.40 (1.15-1.70) and median paired B/A package J/token 0.75 (0.60-0.95); SUPPORTED by H34's rule (tok/s >= 1.20 with its CI above 1.0, J/token <= 0.85 with its CI below 1.0) at 0.6, REFUTED 0.25 (the draft's share of a faster target step grows), INCONCLUSIVE 0.15. | H34's `bench/hetero-spec/h34.sh` and `h34.py`, n = 12 pairs A B / B A, 5 s load window per request, unchanged except as listed in the cross-host pre-registration: cluster `k3dgpu-x-h34`, `K3D_GPU_VENDORS=none`, node container on CPUs 0-23,48-71, crescent amendment C1, `typical` only. Run 2026-10-20 ([H34x run](#h34x-run-xeon-6741p-2026-10-20)): load MATCHED, 12/12 pairs; B/A decode tok/s 0.758 (CI 0.688-0.809) and package J/token 1.241 (CI 1.168-1.347), B slower and costlier in 12/12 pairs (A 37.2, B 28.0 tok/s): the draft slows decode on this Xeon. | REFUTED |
| H25x | H25's hypothesis (T part) on the Xeon 6741P: agents that send a byte-identical system prompt and tools to one OVMS CPU server reuse one prefix KV, so an added agent's first chunk arrives at least 4x sooner with the shipping prefix caching on than with `--enable_prefix_caching=false`. A replication: the original model and prefix shape are kept. | Written 2026-10-20, before any H25x measurement. Added-agent median TTFT OFF 1.2 s (0.5-2.5), ON 0.15 s (0.06-0.4), ratio 8x (4.5-14), the ratio's 95% CI lower bound >= 4x; c/OFF >= 0.8. SUPPORTED 0.75; the risk is fixed per-request overhead, which a 5-10x faster prefill leaves as a larger share of the ON time. M (memory per agent) not measured, as in H25 run 4. | H25's `bench/shared-prefix/run.sh` and `probe.py score` (A3 first-chunk TTFT, A4 amendment-27 load rule), ON OFF interleaved 3+3 rounds, unchanged except as listed in the cross-host pre-registration: cluster `k3dgpu-x-h25`, `K3D_GPU_VENDORS=none`, node container on CPUs 0-23,48-71 (`NODE_CPUS`), crescent amendment C1, `typical` only. Run 2026-10-21 ([H25x run](#h25x-run-xeon-6741p-2026-10-21)): load MATCHED (amendment 27), no round void; added agent's first chunk 0.245 s ON vs 1.724 s OFF (7.05x, CI 6.94-7.13, n = 15 per arm), c/OFF 0.99, every round pair >= 4x: T SUPPORTED; M not measured (A3). | SUPPORTED |
| H28x | H28's hypothesis on the Xeon 6741P: N = 8 agent pods that mmap one 1 GiB model through an image volume share one page-cache copy and become ready in at most half the time of pods that copy it into an emptyDir. A replication (H28 has no model; the 1 GiB blob is kept). | Written 2026-10-20, before any H28x measurement. Memory parts hold (0.95): T PSS <= 160 MiB per pod, model page cache about 1.0x size (T) vs about 9x (C). Time part uncertain: the copy runs at memory speed on this host, so the fixed pod-start cost is a larger share of both arms; T/C round-median ready 0.55 (0.35-0.8). SUPPORTED 0.35, REFUTED (time part) 0.5, other rows 0.15. | H28's `bench/zero-copy-models/probe.sh` (A1, A2), arms C T T C C T plus the unscored warm-up, unchanged except as listed in the cross-host pre-registration: cluster `k3dgpu-x-h28`, crescent amendment C1 passed to the plain `k3d cluster create`, `typical` only, scored by `bench/zero-copy-models/score.py` (H28's table, committed with this pre-registration). | PENDING |
| H7x | H7's hypothesis on the Xeon 6741P, run to H7's full protocol: OVMS's decode lead over EAT's vLLM CPU engine comes from weight precision, not from the engine. One checkpoint (`Qwen/Qwen3-4B` @ `1cfa9a72`), OVMS 2026.4.1 on its 16-bit IR (`OpenVINO/Qwen3-4B-fp16-ov` @ `00298f1c`) and vLLM CPU v0.24.0 in bf16 with EAT's chart defaults decode within 15% of each other, and land within 15% in effective weight bandwidth. A replication: H7's models and versions are kept. | Written 2026-10-20, before any H7x measurement. On AMX both engines compute in bf16 and batch-1 decode reads the same 8.0 GB per token, so the engine gap shrinks: R = OVMS/vLLM decode 1.10 (0.85-1.40); SUPPORTED 0.45, REFUTED 0.30, INCONCLUSIVE 0.25. Reported, not scored: prefill TTFT on ~~\~1936~~ ~1850 tokens (corrected in the 2026-10-21T22:30Z revision to the prompt `h7x.sh` has always sent, 56 copies of one sentence; value unchanged; struck through 2026-10-29) vLLM/OVMS 1.5 (0.7-4; the i9 gap of 4.3x was outside H7's prediction), C/B decode (`--enforce-eager` dropped) 1.0-1.3, decode incremental J/token per arm. | `bench/apb/h7x.sh` (H7's `h7.sh` extended to the pre-registered protocol: H2's workloads and energy method, blocks A B B A A B B A A B, then one C block with `--enforce-eager` dropped), scored by `bench/apb/h7x.py`; decision rule, host changes and hold in [H7x pre-registration](#h7x-pre-registration-h7-to-its-full-protocol-on-the-xeon-6741p-2026-10-20). | PENDING |
| H7x | (Arms B and C superseded by [amendment V1](#h7x-amendment-v1-eat-release-110-for-arms-b-and-c-2026-10-29): EAT release-1.1.0's stock, vLLM CPU v0.28.0; H7's vLLM version is no longer kept.) H7's hypothesis on the Xeon 6741P, run to H7's full protocol: OVMS's decode lead over EAT's vLLM CPU engine comes from weight precision, not from the engine. One checkpoint (`Qwen/Qwen3-4B` @ `1cfa9a72`), OVMS 2026.4.1 on its 16-bit IR (`OpenVINO/Qwen3-4B-fp16-ov` @ `00298f1c`) and vLLM CPU v0.24.0 in bf16 with EAT's chart defaults decode within 15% of each other, and land within 15% in effective weight bandwidth. A replication: H7's models and versions are kept. | Written 2026-10-20, before any H7x measurement. On AMX both engines compute in bf16 and batch-1 decode reads the same 8.0 GB per token, so the engine gap shrinks: R = OVMS/vLLM decode 1.10 (0.85-1.40); SUPPORTED 0.45, REFUTED 0.30, INCONCLUSIVE 0.25. Reported, not scored: prefill TTFT on ~~\~1936~~ ~1850 tokens (corrected in the 2026-10-21T22:30Z revision to the prompt `h7x.sh` has always sent, 56 copies of one sentence; value unchanged; struck through 2026-10-29) vLLM/OVMS 1.5 (0.7-4; the i9 gap of 4.3x was outside H7's prediction), C/B decode (`--enforce-eager` dropped) 1.0-1.3, decode incremental J/token per arm. | `bench/apb/h7x.sh` (H7's `h7.sh` extended to the pre-registered protocol: H2's workloads and energy method, blocks A B B A A B B A A B, then one C block with `--enforce-eager` dropped), scored by `bench/apb/h7x.py`; decision rule, host changes and hold in [H7x pre-registration](#h7x-pre-registration-h7-to-its-full-protocol-on-the-xeon-6741p-2026-10-20). B and C are EAT release-1.1.0's stock (`0c726f57`, vLLM CPU v0.28.0), after a further setup smoke, per [amendment V1](#h7x-amendment-v1-eat-release-110-for-arms-b-and-c-2026-10-29). | PENDING |
| H55 | On the i9-13900T, an SLO gate on the E-core neighbour's `cpu.max`, set from the P-core inference server's own median TTFT, keeps CPU-served prefill within +10% of quiet while the neighbour keeps more throughput than H5's frequency-cap frontier (CMIN: +23.3% at 72.4% neighbour loss) and than the best static quota at the same penalty. | Written 2026-10-20, before any H55 measurement; control rule revised the same day, still before any measurement (see the section). Pre-registered prior, not a result: N0 about +40% (H5 replicate), K50 +20 to +30% at about 50% loss, K25 +8 to +15% at about 75% loss. A_slo climbs only while the penalty at its level is at most +5%, so it is expected to stay at the 25-37.5% levels, close to K25: +8 to +15% at 70-78% loss, failing (b) or (c). Expected verdict REFUTED: package power is the coupling, duty-cycling the neighbour is less power-efficient per op than H5's frequency cap, and a closed-loop c = 1 prefill load leaves no idle gap to admit it into. If H55 holds, A_slo settles at 50% or above at +10% or less. | H5 amendment A1's host-level harness (OVMS 2026.4.1, `OpenVINO/Qwen3-4B-int4-ov`, `bench/prefill-guard/prompt.txt` with a unique tag per request, CPUs 0-14; `stress-ng --cpu 15 --cpu-method matrixprod` on CPUs 17-31; tool proxy on CPU 16), with a 900 s closed-loop prefill window per run (16 tokens, concurrency 1). Arms Q, N0 (`cpu.max max`), K50 (`750000 100000`), K25 (`375000 100000`) and A_slo (`bench/admission/ctl.py` steps 25-100% of 15 CPUs on windows of 20 requests: down if the window median > 1.10 x the round's quiet reference, the median of 20 pre-roll requests; up if it is at most 1.05 x that), 5x5 Latin square, one round per lock hold. Supported only if A_slo's median penalty is at most +10%, its neighbour loss is below 72.4%, and no static arm reaches +10% at a loss no higher than A_slo's plus the neighbour's own N0 run-to-run spread; inconclusive gates and the hold cap in [H55 pre-registration](#h55-pre-registration-slo-gated-admission-of-an-e-core-neighbour-2026-10-20). Not run. A SUPPORTED H55 does not ship a controller by itself: kubelet owns the sandbox pod's `cpu.max` (`limits cpu "1"`). | PENDING |
| H56 | Two measurement holds in disjoint cpuset slices of the Xeon 6741P do not perturb each other: an A1 B24 agent block in slice 0 (CPUs 0-23,48-71) behaves equivalently, in per-attempt time, within-budget success and tasks completed, alone and with a concurrent identical A1 hold in slice 1 (CPUs 24-47,72-95), without an all-core clock drop on slice 0 and with the package below its power limit. It decides whether crescent may run two A1-or-lighter holds at once ([adoption rule](#h56-pre-registration-parallel-slice-holds-on-crescent-2026-10-20)); it is not a k3d-gpu product claim. | Written 2026-10-20, before any run; predictions unchanged by the 2026-10-21 revision, priors updated to its narrower bands. Geometric-mean W/A time ratio 1.04 (0.99-1.09); paired success W - A -0.01 (-0.06 to +0.04); completion W/A 0.97 (0.90-1.00); own-slice MHz W/A 0.97; W package 0.75 PL1 (0.55-0.95), A package 0.5 PL1; time pairs 80 (50-96). Batch-1 int4 decode is DRAM-bound, and the sibling's decode and its stream load share DRAM, L3 and package power. Verdict priors: SUPPORTED 0.30, INCONCLUSIVE 0.45, REFUTED 0.25. | Stage 0 first, read-only: `bench/eatflow/crescent/lockwait.py` over both campaigns' `campaign.log`; if lock wait our slots could remove is under 0.10 of hold time, stage 1 is not run and the row stays PENDING (`h56.sh` refuses through `h56.py gate`). [Amendment S0](#h56-pre-registration-parallel-slice-holds-on-crescent-2026-10-20) (2026-10-21, before any stage-1 hold, stage-0 reading disclosed): the share is pooled lock wait over hold time across every campaign log on crescent, nothing netted out; bar 0.10 unchanged. Then one W wiring smoke, which must show slice-0 setup <= 900 s and >= 3 tasks in 120 s. Stage 1: `bench/eatflow/crescent/h56.sh N` for N = 1..8, order A W W A A W W A, one condition-hold per exclusive campaign lock (slot locks first). A: slice 0 alone, slice 1 idle. W: the sibling in slice 1 first (cap 3000 s, budget 1800 s), slice 0 from the sibling's run budget on (cap 2400 s, budget 720 s, the same as A). Hold pairs (1,2) (4,3) (5,6) (8,7), tasks by id; 90% two-stage bootstrap CIs (hold pairs, then tasks; 10,000, seed 56; TOST at alpha 0.05). SUPPORTED: all 8 positions filled, time CI inside [1/1.05, 1.05], success CI inside [-1/24, +1/24], completion W/A >= 1/1.05, >= 60 time pairs, own-slice MHz W/A >= 0.95, W package < 0.95 PL1 and PL1 read. REFUTED: the time CI wholly outside its band, the success CI wholly outside its band, or the completion CI wholly below 1/1.05. Otherwise INCONCLUSIVE. VOID holds are replaced, at most 2. `bench/eatflow/crescent/h56.py score`. | PENDING |

Expand Down Expand Up @@ -1804,6 +1804,73 @@ bench/xhost/hold.sh H7x 10800 env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61 K
bench/apb/h7x.sh
```

### H7x amendment V1: EAT release-1.1.0 for arms B and C (2026-10-29)

Recorded 2026-10-29, before any H7x hold, while H7x's pre-registration was still unmerged (PR
#196). Only the six setup smokes have run (engines brought up, no request sent, all on vLLM
v0.24.0); they stay as records. No prediction, statistic, decision rule, protocol step, CPU or
memory sizing, or cap changes. What changes is which EAT release arms B and C are, so H7x no
longer keeps H7's vLLM version.

**Why.**

- The pre-registration pins B at EAT `0d3438d` (release-1.0.3-hotfix), EAT's latest release when
it was written. That release's `xeon-values.yaml` names `vllm/vllm-openai-cpu:v0.24.0`, pinned
here through `integrations/eat/vllm-values.yaml`.
- EAT published release-1.1.0 on 2026-10-22 (`0c726f57`). It is still EAT's latest release on
2026-10-29; main is one diagram commit ahead. Its `xeon-values.yaml` moves the image to
`v0.28.0`.
- B is EAT's vLLM CPU engine with EAT's chart defaults, that is, EAT stock. The repository's rule
is that EAT-stock arms track EAT's latest release, as
[crescent amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29)
did for H50 and H51. Run on the replaced release, H7x would measure an engine EAT no longer
ships.

**The change.**

1. B is EAT release-1.1.0's stock: `core/helm-charts/vllm` at `0c726f57`, with `xeon-values.yaml`
plus `integrations/eat/vllm-values.yaml`, KV 6 GiB and `--max-model-len 32768`, as
pre-registered. Its image is `vllm/vllm-openai-cpu:v0.28.0`, pinned to the tag's index digest
`sha256:197354e6475e638eb5ccca09beb078488046a82de0e204e7429bf1a804df9647` (Docker Hub, read
2026-10-29). C is B with `--enforce-eager` removed and `VLLM_CACHE_ROOT=/tmp/vllm`, as
pre-registered. A is unchanged.
2. Runner. `bench/apb/h7x.sh` now sets B's tag itself (`--set image.tag`), so the shared
overlay's own tag no longer reaches H7x. It also refuses a chart whose `xeon-values.yaml`,
`values.yaml` and `templates/deployment.yaml` do not have release-1.1.0's sha256; the run
directory keeps `chart.sha256` and `chart-check.txt`.
3. What changes in B's and C's objects. Both releases were rendered through `h7x.sh` itself
(`H7X_RENDER_ONLY=1`, `H7X_VLLM_MEM=fit`, 16 CPUs, helm v4.1.1). They differ in two places:
the image, and the pod `securityContext`, which gains `fsGroup: 1001` (release-1.1.0's chart
default). The args, the env ConfigMap, the volumes, the probes, the 16-CPU request and the
fit's 22Gi limit are the same. The `0d3438d` render reproduces smoke 6's `vllm.yaml` and
`vllm-c.yaml` byte for byte. A's render does not change.
4. What changes inside the engine. EAT's env still sets `VLLM_CPU_SGL_KERNEL=1`, but vLLM v0.28.0
no longer reads it (vllm-project/vllm#50801, inside v0.28.0). Its CPU dispatch
(`vllm/model_executor/layers/utils.py`) sends bf16 weights of up to 1 MiB to the SGL AMX
kernel and larger ones to oneDNN. Every dense projection of Qwen3-4B is larger than 1 MiB, so
B's linear layers move from the SGL kernel (v0.24.0 with the flag on) to oneDNN. That is part
of the engine EAT now ships. It is disclosed here and not adjusted.
5. Setup smoke 7, before the hold: `H7X_SMOKE=1`, order B C, the run's settings (16 CPUs on
0-7,48-55, `H7X_VLLM_MEM=fit`, `H7X_READY_S=1800`), no request sent, recorded next to smokes
1-6. If B is not Ready, the hold does not run until a further dated amendment. If only C is not
Ready, the run goes ahead as pre-registered, with C/B reported as not measured.
6. Unchanged: A, the models and their commits, the protocol, the decision rule, the host changes,
the node image, and the hold and its cap. The int4 second arm stays not run; it would first
need vLLM v0.28.0 CPU to be shown to load an int4 build of the checkpoint.

**Reproduce.** As the pre-registration's, except that the EAT checkout is release-1.1.0:

```bash
git -C $X/eat-src fetch -q origin && git -C $X/eat-src checkout 0c726f576595971c53cc45b94e08ba969beb4a4a
```

**Predictions.** None is edited. They were written for v0.24.0; this amendment changes the release
B and C run, not what is predicted of them. The run section reads them as written.

**Disclosed.** Written before any H7x data, with the predictions in view. The row no longer
replicates H7's vLLM version. It keeps H7's models, OVMS arm and protocol, and compares OVMS
2026.4.1 with the vLLM CPU engine that EAT ships now.

### Cross-host pre-registration: H25x, H27x, H28x, H34x on the Xeon 6741P (2026-10-20)

Written 2026-10-20T22:20Z, before any H25x, H27x, H28x or H34x measurement. Nothing here is a
Expand Down
18 changes: 17 additions & 1 deletion bench/apb/h7x.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,13 @@
#
# Environment:
# APB_MODELS dir with OpenVINO/Qwen3-4B-fp16-ov and Qwen/Qwen3-4B (bench/xhost/fetch_models.py) (required)
# APB_EAT_CHART EAT checkout's core/helm-charts/vllm at 0d3438d (required)
# APB_EAT_CHART EAT checkout's core/helm-charts/vllm at release-1.1.0, 0c726f57 (required;
# checked by sha256, EXPERIMENTS.md H7x amendment V1)
# APB_OUT run directory parent (default ./apb-runs)
# H7X_CPUS CPUs each engine is sized to (default 16, H7's P-core balloon)
# H7X_NODE_CPUS cpuset for the node container after `up` (default none)
# H7X_ORDER block order (default "A B B A A B B A A B C")
# H7X_CHART_CHECK 0 skips the release-1.1.0 chart check (tests only; default 1)
# H7X_VLLM_MEM memory limit for B and C (default: the overlay's 16Gi; `fit`: k3d-gpu fit's
# weights + runtime for this checkpoint, plus the row's 6 GiB KV, x 1.1, rounded up)
# K3D_GPU_CLUSTER cluster name (default apb-h7x); K3D_GPU_* pass through to `up`
Expand Down Expand Up @@ -62,6 +64,19 @@ stage() { # stage <local dir> <path under /var/apb-models>
}

stamp render
# B and C are EAT release-1.1.0's stock (H7x amendment V1): its chart, checked here by the sha256
# of the files that shape them, and its xeon-values.yaml image tag pinned to the tag's index
# digest. The tag is set here, not by integrations/eat/vllm-values.yaml, whose own tag is not used.
B_TAG=v0.28.0@sha256:197354e6475e638eb5ccca09beb078488046a82de0e204e7429bf1a804df9647
if [ "${H7X_CHART_CHECK:-1}" != 0 ]; then
printf '%s %s\n' \
05c1e55b0864e6b576cc84e7ba330f45021f855e84f677f4e68add24d15a4da0 xeon-values.yaml \
636ff3968e4983cdd9cfbb9372220674921a4c725483a7744527842370e9182e values.yaml \
93ef991af65dc4f448b504b9934131ee9acb59135962bab6103faf35ab1a226d templates/deployment.yaml \
> "${OUT}/chart.sha256"
(cd "${CHART}" && sha256sum -c "${OUT}/chart.sha256") > "${OUT}/chart-check.txt" 2>&1 \
|| { echo "${CHART} is not EAT release-1.1.0's chart:"; cat "${OUT}/chart-check.txt"; exit 1; }
fi
# H7's OVMS arm has no draft (it predates the CPU Fast Draft default of #116).
# H7 also predates the host fit (#174), which would size A's KV pool to this host; K3D_GPU_FIT=off
# keeps the manifest as H7 rendered it.
Expand All @@ -82,6 +97,7 @@ print("%dGi" % math.ceil((d["host_bytes"] - d["kv_bytes"] + 6 * 2**30) * 1.1 / 2
fi
python3 "${HERE}/stage.py" values "${CHART}/xeon-values.yaml" "${REPO}/integrations/eat/vllm-values.yaml" 6 "${OUT}/values.yaml"
helm template vllm-qwen-3-4b-cpu "${CHART}" --values "${OUT}/values.yaml" --set LLM_MODEL_ID=Qwen/Qwen3-4B --set cpu="${CPUS}" ${VLLM_MEM:+--set memory="${VLLM_MEM}"} \
--set image.tag="${B_TAG}" \
--set global.monitoring=false --set svcmonitor.enabled=false --set ingress.enabled=false --set route.enabled=false > "${OUT}/eat.yaml"
python3 "${HERE}/stage.py" vllm "${OUT}/eat.yaml" "${OUT}/vllm.yaml" Qwen3-4B 32768
# C: the same Deployment and Service, renamed, with --enforce-eager removed (the only change).
Expand Down
Loading