Skip to content

bench(h59): the H59-family runner (amendment R1); H57d amendment R2, Od renders in the hold's cluster - #268

Open
88plug wants to merge 2 commits into
mainfrom
claude/h59-runner
Open

88plug wants to merge 2 commits into
mainfrom
claude/h59-runner

Conversation

@88plug

@88plug 88plug commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

The H59, H59d and H59w pre-registration (#197) needs a runner amendment with tests merged before the first H59-family hold. This is that amendment. It also fixes the H57d runner's Od render for the H57w rows that have not run.

H59 runner (EXPERIMENTS.md "H59 amendment R1")

bench/h59/:

  • run.py: the arm table, the hold orders, and host profiles for crescent (Xeon 6741P, H59), the desktop (i9-13900T, H59d) and wildct (i7-13700, H59w).
  • Each arm is rendered with k3d-gpu agent manifest --model in the hold's cluster. K3D_GPU_CLUSTER is set, revisions are pinned with K3D_GPU_AGENT_REVISION, and inherited agent variables are dropped.
  • The render is staged against the campaign's model directory and checked. An arm whose render is not the registered one (images, model paths, flags, the P-core balloon) does not start.
  • run.py also writes crescent's load verdict under H54's void rule, for h59.py --load GATE.json, and runs the prefix-reuse phase (H25's shape).
  • hold.sh, hold-run.sh and arm.sh run the registered holds:
    • holds 1-3: N=1, N=4, N=16 (crescent only), TTFT, prefix reuse, in the rotated orders (O, L), (L, O), (O, L);
    • hold 4: quality, in (L, O);
    • spares 5 and 6, each repeating a failed hold.
  • Each hold runs once, under the host's own lock, start rule, floors and cap.
  • The records use the layout bench/h59/h59.py (ledger(H59, H59d, H59w): pre-register engine choice per CPU class #197 S1/S2) scores; the scorer is unchanged.
  • prep.sh downloads the pinned files and checks them against the Hub's sha256. It builds the drafter by running the render's own agent: llama.cpp backend the fit picks, and busy Intel cards left out of the device plugin #182 conversion Job once.
  • bench/wildct/row.sh gains h59w and h59w-prep.

The amendment also records:

H57d runner fix (EXPERIMENTS.md "H57d amendment R2")

bench/h57d/arm.sh rendered Od without K3D_GPU_CLUSTER. Since #183 that render goes through context k3d-gpu, which a hold's kubeconfig does not hold. So Od lost its P-core label and got 1 CPU, which is H57dp's deviation in #247. Od's render now gets the hold's cluster. The amendment governs the H57w rows on wildct, which have not run.

Reported for the owner, not acted on: the desktop load gate

On the H57d records and the H57dp records (#247), the foreign P-core load read over the N=4 phases moves with the arm, not with the host:

  • during OVMS's phases the host's other work leaves the P-cores;
  • during the llama.cpp arms' phases (8 decode threads) it stays.

In the arms' 60 s idle windows the arms match: H57d spread 1.46, H57dp 0.21, against the 2.00 tolerance. Over N=4 they do not: 2.61 and 2.70. That is the post-treatment trap H54 amendment R4 fixed on crescent.

Changing the H59d/H59w gate is the owner's decision. No gate is changed here. The runner records both readings.

Tests

  • bench/h59/test_run.py (new, in CI, 33 tests, about 3 min):
    • renders both arms through the real launcher against the fit fixtures, with a kubectl stand-in that answers only for the hold's context;
    • runs arm.sh against a local stub server for four desktop holds, then scores them with h59.py end to end (SUPPORTED on a 2x-faster L);
    • covers crescent's N=16 phase and void re-run, the gate branches, the prefix phase, hold.sh's guards and the shim.
    • checks the stub server's request log against the registered counts and settings (warmup, N=1, N=4, TTFT, prefix, GSM8K, the 900 s boot limit);
    • refuses a sibling CPU SKU (the exact model number the engine rule matches), a spare 6 before spare 5 or over a measured spare 5, and a hold that started but recorded no rc.
  • bench/h57d/test_h57d.py: renders Od through arm.sh and reads the P-core label and 16 CPUs. It failed before the fix.
  • tests/wildct_hold_test.sh: the h59w rows.

Run locally:

  • bench/h59/test_run.py, test_h59.py, bench/h57d/test_h57d.py, bench/h54/test_h54.py, bench/wildct/test_wildct.py: OK;
  • tests/wildct_hold_test.sh: 41 passed;
  • tests/ci_wiring_test.sh: 70 passed;
  • tests/claims_check.sh: clean;
  • scripts/images-lock check and tests/image_pins_test.sh: pass (the test file holds no tag-only image string);
  • tests/ledger_drift_check.py: agrees;
  • console/ledger-rules.json: matches;
  • shellcheck v0.11.0 at warning: clean.

No prediction, hypothesis or verdict cell changed. Only the method column of H59, H59d, H59w and H57wa gained a link. No host was touched and no hold ran.

…d renders in the hold's cluster)

The H59, H59d and H59w pre-registration requires a runner amendment with tests merged before
the first hold. This adds it.

bench/h59/run.py holds the arm table, the hold orders, and host profiles for crescent (Xeon
6741P), the desktop (i9-13900T) and wildct (i7-13700). It renders each arm with
`k3d-gpu agent manifest --model` in the hold's cluster (K3D_GPU_CLUSTER set, revisions pinned,
inherited agent variables dropped), stages it against the campaign's model directory, and
refuses an arm whose render is not the registered one: images, model paths, flags, the P-core
balloon. It also writes crescent's load verdict under H54's void rule and runs the prefix-reuse
phase. hold.sh, hold-run.sh and arm.sh run the registered holds: performance holds 1-3 at
N=1, N=4 (and N=16 on crescent), TTFT and prefix reuse, the quality hold, and the spares. Their
records are in the layout bench/h59/h59.py scores. prep.sh downloads the pinned model files and
builds the drafter with the render's own conversion Job. bench/wildct/row.sh gains h59w and
h59w-prep.

bench/h57d/arm.sh rendered Od without K3D_GPU_CLUSTER. Since #183 that renders through context
k3d-gpu, which a hold's kubeconfig does not hold, so Od lost its P-core label and got 1 CPU
(H57dp's deviation). It now passes the hold's cluster. H57d amendment R2 records the fix for the
H57w rows, which have not run.

EXPERIMENTS.md: H59 amendment R1 (mechanics only: the runner, the flags read at each engine's
pin, staging, holds, host profiles, the engine rule as merged), and a report for the owner on
what the desktop load gate reads. On the H57d and H57dp records, the foreign P-core load over the
N=4 phases moves with the arm, not the host, while the arms' idle windows match. No gate is
changed. H57d amendment R2. Links from the H59, H59d, H59w and H57wa rows; no prediction changes.

Tests: bench/h59/test_run.py renders both arms through the real launcher against the fit
fixtures, with a kubectl stand-in that answers only for the hold's context. It runs arm.sh
against a local stub server and scores four desktop holds with h59.py end to end (SUPPORTED on a
2x-faster L). It also covers crescent's N=16 phase and void re-run, the gate, the prefix phase,
hold.sh's guards and the shim. bench/h57d/test_h57d.py renders Od through arm.sh and reads the
P-core label and 16 CPUs; it failed before the fix. tests/wildct_hold_test.sh covers the h59w
rows. The new suite is wired into CI.
88plug added a commit that referenced this pull request Oct 30, 2026
A ledger line listed only the rows whose ledger sections name its path, so a
runner another row runs or imports turned historical when its own rows finished
while the reusing row was still PENDING. #262 landed the H57dp and H57de
verdicts and bench/h57d/** went historical while wildct's H57w rows still run
it through bench/wildct/row.sh; the guard then refused #268's H57d amendment R2
edit of bench/h57d/arm.sh as a historical edit.

A scan of every ledger runner's code for the bench/ paths it calls found the
reuses, and each line now lists the reusing rows and their amendment prefix:
bench/h57d (the H57w rows; h57d.py also H59), bench/h54/h54.py with its tests
and GSM8K items (H59, H60), bench/prefill-guard's cpufreq.sh and prompt.txt
(H55), crescent's serving.py (the cross-host rows, H60, H59),
bench/xhost/fetch_models.py (the wildct rows), bench/load (the desktop,
crescent and cross-host rows and H59w) and bench/shared-prefix (H59). The rest
of each tree keeps its own rows, and the H57dp and H57de records get their own
historical line now that the H57d tree is an arm again.

tests/pin_classes_test.py checks each reuse: the reusing file still names the
path, the path's line lists the row, and the path is an arm while the row is
PENDING.
…ted holds, pinned test images

The images.lock check failed on three tag-only image strings in bench/h59/test_run.py. They are
now derived from the pinned references in run.py and the test's node image, so no unpinned
literal is left in the file.

hold.sh compared the host's CPU by substring, so an i7-13700K passed as the i7-13700 and an
i9-13900TE as the i9-13900T. The engine rule matches a row by the exact model number
(console/model_opt.py class_tokens), so the host check now uses that function (run.py cpu).

A spare could repeat a hold that spare 5 had already measured, and h59.py reads spare 6 over
spare 5, so a measured record could be replaced. Spare 6 now runs only after spare 5, and a
spare repeats a hold only while the record standing for it has an arm not measured.

A hold interrupted before it recorded its rc could run again under the same number. hold.sh now
writes `started` once the floors pass, refuses a hold that has one, and spare-ok counts it as run.

test_run.py now checks the server's request log against the registered counts and settings
(3 warmup, 8 at N=1, 16 at N=4, 4 TTFT requests at 16 tokens on H54's prompt, 6 prefix agents,
GSM8K 0-49 at 1024 tokens with thinking off, unique prompts, the 900 s boot limit). Before this a
runner with 6 N=1 requests, 32-token TTFT requests or a 600 s boot limit passed every test. The
new guard tests failed on the previous commit.

Amendment R1 records the spare order, the `started` file and the CPU class rule, and cites
OVMS's `--task` option at src/cli_parser.cpp:324, where it is defined (:62 reads it). No
prediction, statistic, gate, cap or hold count changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant