Repository navigation
bench: newest releases for the bench arms the ledger lets move; H7/H8 pin their own vLLM image - #237
Merged
Merged
Conversation
Three bench pins are older than their project's newest release on purpose. Each now says why next to the pin: - h54.py's llama.cpp b11749 feeds H54q, H57dp/H57de and H57wa/wm/wb, which pair with or replicate b11749 records. Moving it to v0.8.0 (b11960) needs a dated amendment for each of those rows first. - live.sh's A3/A3ref and apb/arm.sh's vllm-cuda arm run vLLM CUDA v0.31.0, the image the done rows ran. No pending row runs them, and the pin is kept so those rows reproduce. Comments only; nothing a run reads changes.
Both scripts took B's image tag from the shared overlay integrations/eat/vllm-values.yaml, which still names v0.24.0. EAT's latest release ships v0.28.0, and that overlay is due to move with it. When it does, H7's and H8's scripts would start rendering a different engine from the one their run headers record. Each script now sets the tag those rows ran (v0.24.0 by index digest) on the helm command line, after the overlay. Rendered from EAT 0d3438d's chart, the output is byte-identical to before, and it stays on v0.24.0 when the overlay's tag is moved to v0.28.0. Test: bench_test.sh checks both scripts carry the pin; it failed before this change.
The reason recorded next to h54.py's llama.cpp pin listed H57wb among the rows that depend on b11749. H57wb compares k3d-gpu auto (OVMS) with EAT stock vLLM CPU and runs no llama.cpp arm. The rows that do take this pin are H54q (Lg, paired with H54a's b11749 records under amendment Q1) and, through bench/h57d/h57d.py, H57dp, H57de, H57wa and H57wm. Comment only; nothing a run reads changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR moves the bench arms to the newest releases where the ledger allows it. Where a pin has to stay, the reason is now written next to it. Every version below was re-read on 2026-10-29 from the registry or the project's releases:
0c726f57)8642541b, buildb11960Every new or kept digest in these changes resolves with
docker buildx imagetools inspect.Bumps, each with a dated amendment, stacked on the unmerged pre-registrations
These two rows have not run, and their pre-registrations are not on main yet, so the amendments target those PRs' branches:
server-b11783toserver-b11960(v0.8.0,sha256:8052c13e…). Checked before the bump:llama-server --helpoutputs differ only by a router-mode flag.qwen4exploader reads the same GGUF keys. It falls back to the separate tensors, and the pinned Flash-Next GGUF carries only those (its two shard headers were read).v0.28.0@sha256:197354e6…and the chart at0c726f57.h7x.shnow pins the tag itself and refuses any chart that is not release-1.1.0's (checked by sha256).h7x.sh, the two releases differ only in the image and the chart'sfsGroup: 1001.VLLM_CPU_SGL_KERNEL, so B's dense bf16 projections run on oneDNN.Merge each into its base before that pre-registration lands.
This PR (main)
bench/apb/h7.shandbench/apb/arm.sh(eat-vllm-cpu) pin v0.24.0, the image H7 and H8 ran. Both took the tag fromintegrations/eat/vllm-values.yaml, which is due to move to EAT's v0.28.0. After this PR, that move no longer changes what H7's and H8's scripts reproduce.0d3438d's chart are byte-identical before and after the change.bench_test.shcheck covers the pin. It failed before the change.h54.py's llama.cpp b11749: H54q (amendment Q1), H57dp/H57de and H57wa/wm/wb pair with or replicate b11749 records. A newer build needs a dated amendment for each of those rows first.live.sh(A3/A3ref) andapb/arm.sh(vllm-cuda): only done rows ran them, and the pin is kept so those rows reproduce.Kept as they are
h54.py's vLLM CPU v0.32.0, OVMS 2026.4.1 and SGLang v0.5.22-xeon.live.shB2 and LiteLLM, B3 2025.4, and the pairs/apb hold wrappers' v0.28.0. These match EAT release-1.1.0.agent.lock's H40-H48 entries.live.sh's B1 default (v0.24.0).h54.py'svllm024(B1g).full-b11749, the drafter converter that the committed weights' sha256 depends on.probe.sh's python:3.13-slim, the image of H28's measured pods, which H28x and H28w replicate.Coordination
bench/eatflow/agent.lock(proxy_python, redis and the redis read date) andbench/zero-copy-models/probe.sh's PYIMG. The second one is the measured pod image of the pending H28x and H28w, so either nightly: repin the moved digests and keep the digest PR out of .github/workflows/ #220 drops those files or the two rows get a dated amendment before they run.scripts/images-lock repinscansbench/and will re-pin any bench pin whose tag the registry re-points, pending rows included.python:3.14.8-slim→3.14.10-slim@sha256:3cabfd7d…) and the H53 card placeholder (busybox:1.37→1.38.0@sha256:a46257e0…) inlive.shmove in a follow-up on top of nightly: repin the moved digests and keep the digest PR out of .github/workflows/ #220. That follow-up carries an H53 mechanics amendment. BusyBox labels 1.38.0 "unstable", as it labelled 1.37.0; its stable line is 1.36.1.integrations/eat/vllm-values.yaml(integrations area) can move to v0.28.0 once this PR and ledger(H7x): amendment V1, arms B and C on EAT release-1.1.0 (vLLM CPU v0.28.0) #235 are in. After that, no bench row reads its tag.Tests run
tests/bench_test.sh: 59 passed.scripts/images-lock check.tests/image_pins_test.sh: 39 passed.tests/claims_check.sh.tests/ledger_drift_check.py.bench/eatflow(pairs, solo, agentbench, crescent holds),bench/h54,bench/h57dandbench/apb(apb, stage).