Skip to content

bench: newest releases for the bench arms the ledger lets move; H7/H8 pin their own vLLM image - #237

Merged
88plug merged 3 commits into
mainfrom
claude/latest-bench-arms-1029
Oct 29, 2026
Merged

88plug merged 3 commits into
mainfrom
claude/latest-bench-arms-1029

Conversation

@88plug

@88plug 88plug commented Oct 29, 2026

Copy link
Copy Markdown
Owner

This PR moves the bench arms to the newest releases where the ledger allows it. Where a pin has to stay, the reason is now written next to it. Every version below was re-read on 2026-10-29 from the registry or the project's releases:

Project Newest release on 2026-10-29 Where it was read
vLLM v0.32.0 PyPI and GitHub; v0.32.1rc0 and v0.33.0rc0/rc1 are prereleases
EAT release-1.1.0 (0c726f57) GitHub; main is one diagram commit ahead
llama.cpp v0.8.0, commit 8642541b, build b11960 GitHub; b11962 to b11981 are prereleases
OVMS 2026.4.1 registry
SGLang v0.5.22-xeon registry
stress-ng V0.22.01 GitHub

Every new or kept digest in these changes resolves with docker buildx imagetools inspect.

Bumps, each with a dated amendment, stacked on the unmerged pre-registrations

These two rows have not run, and their pre-registrations are not on main yet, so the amendments target those PRs' branches:

Merge each into its base before that pre-registration lands.

This PR (main)

  • bench/apb/h7.sh and bench/apb/arm.sh (eat-vllm-cpu) pin v0.24.0, the image H7 and H8 ran. Both took the tag from integrations/eat/vllm-values.yaml, which is due to move to EAT's v0.28.0. After this PR, that move no longer changes what H7's and H8's scripts reproduce.
    • Renders from EAT 0d3438d's chart are byte-identical before and after the change.
    • They stay on v0.24.0 when the overlay's tag is moved to v0.28.0.
    • A new bench_test.sh check covers the pin. It failed before the change.
  • Reasons recorded next to the held pins (comments only):
    • h54.py's llama.cpp b11749: H54q (amendment Q1), H57dp/H57de and H57wa/wm/wb pair with or replicate b11749 records. A newer build needs a dated amendment for each of those rows first.
    • The vLLM CUDA v0.31.0 arms in live.sh (A3/A3ref) and apb/arm.sh (vllm-cuda): only done rows ran them, and the pin is kept so those rows reproduce.

Kept as they are

  • Already at the newest release:
    • h54.py's vLLM CPU v0.32.0, OVMS 2026.4.1 and SGLang v0.5.22-xeon.
    • stress-ng V0.22.01.
    • The EAT-stock pins H50, H50o, H51, H52b and H53 use: live.sh B2 and LiteLLM, B3 2025.4, and the pairs/apb hold wrappers' v0.28.0. These match EAT release-1.1.0.
  • Records of runs that are done (class c):
    • agent.lock's H40-H48 entries.
    • live.sh's B1 default (v0.24.0).
    • h54.py's vllm024 (B1g).
    • The H54b/H54c GPU images.
    • The C1/C2 desktop arms.
  • Pins that pending rows replicate:
    • full-b11749, the drafter converter that the committed weights' sha256 depends on.
    • The k3d-gpu 0.1.61 and 0.1.63 node images, under E2 item 1 and H54 R1.
    • probe.sh's python:3.13-slim, the image of H28's measured pods, which H28x and H28w replicate.

Coordination

Tests run

  • tests/bench_test.sh: 59 passed.
  • scripts/images-lock check.
  • tests/image_pins_test.sh: 39 passed.
  • tests/claims_check.sh.
  • tests/ledger_drift_check.py.
  • The bench unit suites in bench/eatflow (pairs, solo, agentbench, crescent holds), bench/h54, bench/h57d and bench/apb (apb, stage).
  • shellcheck at warning severity on every changed script.

Three bench pins are older than their project's newest release on
purpose. Each now says why next to the pin:

- h54.py's llama.cpp b11749 feeds H54q, H57dp/H57de and H57wa/wm/wb,
  which pair with or replicate b11749 records. Moving it to v0.8.0
  (b11960) needs a dated amendment for each of those rows first.
- live.sh's A3/A3ref and apb/arm.sh's vllm-cuda arm run vLLM CUDA
  v0.31.0, the image the done rows ran. No pending row runs them, and
  the pin is kept so those rows reproduce.

Comments only; nothing a run reads changes.
Both scripts took B's image tag from the shared overlay
integrations/eat/vllm-values.yaml, which still names v0.24.0. EAT's
latest release ships v0.28.0, and that overlay is due to move with it.
When it does, H7's and H8's scripts would start rendering a different
engine from the one their run headers record.

Each script now sets the tag those rows ran (v0.24.0 by index digest)
on the helm command line, after the overlay. Rendered from EAT 0d3438d's
chart, the output is byte-identical to before, and it stays on v0.24.0
when the overlay's tag is moved to v0.28.0.

Test: bench_test.sh checks both scripts carry the pin; it failed before
this change.
The reason recorded next to h54.py's llama.cpp pin listed H57wb among
the rows that depend on b11749. H57wb compares k3d-gpu auto (OVMS) with
EAT stock vLLM CPU and runs no llama.cpp arm. The rows that do take this
pin are H54q (Lg, paired with H54a's b11749 records under amendment Q1)
and, through bench/h57d/h57d.py, H57dp, H57de, H57wa and H57wm.

Comment only; nothing a run reads changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant