From 6ff68964d51b25c446d59aad5bec8cb815ac7a37 Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Thu, 29 Oct 2026 17:44:48 -0700 Subject: [PATCH 1/2] bench(h59): the H59-family runner, amendment R1; H57d amendment R2 (Od renders in the hold's cluster) The H59, H59d and H59w pre-registration requires a runner amendment with tests merged before the first hold. This adds it. bench/h59/run.py holds the arm table, the hold orders, and host profiles for crescent (Xeon 6741P), the desktop (i9-13900T) and wildct (i7-13700). It renders each arm with `k3d-gpu agent manifest --model` in the hold's cluster (K3D_GPU_CLUSTER set, revisions pinned, inherited agent variables dropped), stages it against the campaign's model directory, and refuses an arm whose render is not the registered one: images, model paths, flags, the P-core balloon. It also writes crescent's load verdict under H54's void rule and runs the prefix-reuse phase. hold.sh, hold-run.sh and arm.sh run the registered holds: performance holds 1-3 at N=1, N=4 (and N=16 on crescent), TTFT and prefix reuse, the quality hold, and the spares. Their records are in the layout bench/h59/h59.py scores. prep.sh downloads the pinned model files and builds the drafter with the render's own conversion Job. bench/wildct/row.sh gains h59w and h59w-prep. bench/h57d/arm.sh rendered Od without K3D_GPU_CLUSTER. Since #183 that renders through context k3d-gpu, which a hold's kubeconfig does not hold, so Od lost its P-core label and got 1 CPU (H57dp's deviation). It now passes the hold's cluster. H57d amendment R2 records the fix for the H57w rows, which have not run. EXPERIMENTS.md: H59 amendment R1 (mechanics only: the runner, the flags read at each engine's pin, staging, holds, host profiles, the engine rule as merged), and a report for the owner on what the desktop load gate reads. On the H57d and H57dp records, the foreign P-core load over the N=4 phases moves with the arm, not the host, while the arms' idle windows match. No gate is changed. H57d amendment R2. Links from the H59, H59d, H59w and H57wa rows; no prediction changes. Tests: bench/h59/test_run.py renders both arms through the real launcher against the fit fixtures, with a kubectl stand-in that answers only for the hold's context. It runs arm.sh against a local stub server and scores four desktop holds with h59.py end to end (SUPPORTED on a 2x-faster L). It also covers crescent's N=16 phase and void re-run, the gate, the prefix phase, hold.sh's guards and the shim. bench/h57d/test_h57d.py renders Od through arm.sh and reads the P-core label and 16 CPUs; it failed before the fix. tests/wildct_hold_test.sh covers the h59w rows. The new suite is wired into CI. --- .github/workflows/ci.yml | 3 + EXPERIMENTS.md | 272 +++++++++++++- bench/h57d/arm.sh | 6 +- bench/h57d/test_h57d.py | 70 ++++ bench/h59/arm.sh | 203 +++++++++++ bench/h59/hold-run.sh | 27 ++ bench/h59/hold.sh | 121 +++++++ bench/h59/k3d-shim.sh | 13 + bench/h59/prep.sh | 64 ++++ bench/h59/reproduce.sh | 36 ++ bench/h59/run.py | 563 +++++++++++++++++++++++++++++ bench/h59/test_run.py | 729 ++++++++++++++++++++++++++++++++++++++ bench/wildct/row.sh | 22 +- tests/wildct_hold_test.sh | 25 ++ 14 files changed, 2148 insertions(+), 6 deletions(-) create mode 100644 bench/h59/arm.sh create mode 100644 bench/h59/hold-run.sh create mode 100644 bench/h59/hold.sh create mode 100644 bench/h59/k3d-shim.sh create mode 100644 bench/h59/prep.sh create mode 100644 bench/h59/reproduce.sh create mode 100644 bench/h59/run.py create mode 100644 bench/h59/test_run.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 54bacd6d..150ecb45 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -269,6 +269,9 @@ jobs: - run: python3 bench/h54gpu/test_h54gpu.py - run: python3 bench/h57d/test_h57d.py - run: python3 bench/h59/test_h59.py + # The H59-family runner (amendment R1): renders through the launcher against the fit + # fixtures, arm.sh against a local stub server, its records scored by h59.py. + - run: python3 bench/h59/test_run.py - run: python3 bench/admission/test_ctl.py - run: python3 bench/admission/test_analyze.py - run: bash bench/admission/test_run.sh diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 1d944d2f..75b82202 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -88,7 +88,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H25w | H25's hypothesis (T part) on wildct: agents that send a byte-identical system prompt and tools to one OVMS CPU server reuse one prefix KV, so an added agent's first chunk arrives at least 4x sooner with the shipping prefix caching on than with `--enable_prefix_caching=false`. A replication: original model, prefix shape, probe, scorer and thresholds. | Written 2026-10-20, before any H25w measurement. Added-agent median TTFT OFF 10 s (7-15), ON 1.0 s (0.6-1.6), ratio 10x (6-14), CI lower bound >= 4x; c/OFF >= 0.8. SUPPORTED 0.8. | H25's `bench/shared-prefix/run.sh` and `probe.py score` (A3 first-chunk TTFT, A4 amendment-27 load rule), ON OFF interleaved 3+3 rounds at `typical`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h25`, the GPU opt-out, rule W2 (load on E-cores 20-23 only), W3, W4, new seeds (`SEED_BASE=2560`). | PENDING | | H28w | H28's hypothesis on wildct: N = 8 agent pods that mmap one 1 GiB model through an image volume share one page-cache copy and become ready in at most half the time of pods that copy it into an emptyDir. A replication (H28 has no model; the 1 GiB blob is kept). | Written 2026-10-20, before any H28w measurement. Memory parts hold (0.95): T PSS <= 160 MiB per pod, model page cache about 1.0x size (T) vs about 9x (C). Time part: T/C round-median ready 0.5 (0.4-0.65), against H28's 0.47 on the i9. SUPPORTED 0.45, REFUTED (time part) 0.4, INCONCLUSIVE or not run (disk) 0.15. | H28's `bench/zero-copy-models/probe.sh` (A1, A2), arms C T T C C T plus the unscored warm-up, `typical` only, scored by `bench/zero-copy-models/score.py`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h28`, node on the P-cores (`NODE_CPUS=0-15`), W2, W3 (passed to the plain `k3d cluster create`), and a disk gate (W6). | PENDING | | H34w | H34's hypothesis on wildct: same-engine CPU Fast Draft (OVMS 2026.4.1, `OpenVINO/Qwen3-0.6B-int4-ov` draft on the CPU) beats the `OpenVINO/Qwen3-4B-int4-ov` target alone on single-stream decode, both on the 16-CPU P-core balloon, under matched background load. A replication: original models, runner, scorer and rule. | Written 2026-10-20, before any H34w measurement. At `typical`: median paired B/A decode tok/s 1.35 (1.15-1.6) and median paired B/A package J/token 0.76 (0.6-0.92). SUPPORTED by H34's rule at 0.6, REFUTED 0.25, INCONCLUSIVE 0.15. | H34's `bench/hetero-spec/h34.sh` and `h34.py`, n = 12 pairs A B / B A, 5 s load window per request, `typical` only, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h34`, the GPU opt-out in place of H34's `intel,hybrid`, W2, W3, W4. | PENDING | -| H57wa | H57da's hypothesis on wildct (i7-13700, the 16 P-core CPUs, AVX2 and AVX-VNNI, no AVX-512, no AMX): with the Gemma 4 26B-A4B QAT family, k3d-gpu's own CPU pick Od (`agent manifest` auto on the P-cores: OVMS 2026.4.1, `OpenVINO/gemma-4-26b-a4b-it-int4-ov`) is the fastest single-stream arm: no other gate-passing arm (llama.cpp with Google's QAT Q4_0 GGUF, with or without the QAT MTP drafter) decodes at N=1 5% or more faster. A cross-host replication of H57da on a second AMX-less client. | Written 2026-10-20, before any H57d or H57w measurement. N=1 median decode tok/s: Od in [7, 18] (0.5); Ld in [10, 26] (0.5); Ld/Od in [1.1, 1.8] (0.5); every arm's GSM8K 0-49 score >= 0.80 (0.6). REFUTED (0.55), SUPPORTED (0.25), INCONCLUSIVE (0.2). Not gated: TTFT on a 1,000-token prompt, aggregate tok/s at N=4, package J/token. | H57da's protocol, runner, statistic and outcome rules (the H57d pre-registration and its runner amendment), unchanged except as listed in the wildct pre-registration: H57d's runner (amendment R1) with the host knobs listed there: clusters `k3dgpu-w-h57d-`, the GPU opt-out in place of `K3D_GPU_VENDORS=hybrid`, W2-W6, hold-start `load1` <= 12, models downloaded on wildct at H57d's revisions, at most 5 holds of at most 75 min. | PENDING | +| H57wa | H57da's hypothesis on wildct (i7-13700, the 16 P-core CPUs, AVX2 and AVX-VNNI, no AVX-512, no AMX): with the Gemma 4 26B-A4B QAT family, k3d-gpu's own CPU pick Od (`agent manifest` auto on the P-cores: OVMS 2026.4.1, `OpenVINO/gemma-4-26b-a4b-it-int4-ov`) is the fastest single-stream arm: no other gate-passing arm (llama.cpp with Google's QAT Q4_0 GGUF, with or without the QAT MTP drafter) decodes at N=1 5% or more faster. A cross-host replication of H57da on a second AMX-less client. | Written 2026-10-20, before any H57d or H57w measurement. N=1 median decode tok/s: Od in [7, 18] (0.5); Ld in [10, 26] (0.5); Ld/Od in [1.1, 1.8] (0.5); every arm's GSM8K 0-49 score >= 0.80 (0.6). REFUTED (0.55), SUPPORTED (0.25), INCONCLUSIVE (0.2). Not gated: TTFT on a 1,000-token prompt, aggregate tok/s at N=4, package J/token. | H57da's protocol, runner, statistic and outcome rules (the H57d pre-registration and its runner amendment), unchanged except as listed in the wildct pre-registration: H57d's runner (amendment R1) with the host knobs listed there: clusters `k3dgpu-w-h57d-`, the GPU opt-out in place of `K3D_GPU_VENDORS=hybrid`, W2-W6, hold-start `load1` <= 12, models downloaded on wildct at H57d's revisions, at most 5 holds of at most 75 min. Od's render names the hold's cluster: [H57d amendment R2](#h57d-amendment-r2-ods-render-names-the-holds-cluster-2026-10-29) (2026-10-29, before any H57w hold). | PENDING | | H57wm | H57dm's hypothesis on wildct: llama.cpp Ld-mtp (Ld plus Google's QAT MTP assistant converted to Q4_0 GGUF, `--spec-type draft-mtp`, default draft length) decodes at N=1 at least 1.20x Ld with the quality gate passed. | Written 2026-10-20. Ld-mtp/Ld in [0.8, 1.3] (0.6). REFUTED (0.65), INCONCLUSIVE (0.15), SUPPORTED (0.2). | As H57wa (same holds and records); H57dm's statistic and rule. | PENDING | | H57wb | H57db's hypothesis on wildct: inside EAT's banking flow (B24) with Gemma 4 26B-A4B QAT, k3d-gpu auto on the P-cores delivers far more tasks within budget than EAT stock CPU (vLLM CPU bf16 on the QAT bf16 weights). | Written 2026-10-20: not runnable here as registered (disk and memory), so INCONCLUSIVE (not run) is expected (0.95). | H57db's design and gates; runs only if at a hold's start / has >= 15 + 70 GB available (51.6 GB of bf16 weights plus images) and MemAvailable >= 100 GiB; otherwise INCONCLUSIVE (not run), with both readings recorded and no bf16 download. | PENDING | | F2a | Open WebUI in a k3d-gpu cluster reaches a public remote endpoint by name: `k3d-gpu quickstart --remote https://gaudi.farmgpu.net` (no hostAliases) gives a chat through Open WebUI's own API that returns. An earlier run saw `Temporary failure in name resolution` for gaudi.farmgpu.net from a pod. | Pre-registered 2026-10-17, before measuring: the in-cluster failure is a cluster DNS gap (CoreDNS forwarding to an upstream the node container cannot use), not the endpoint; once fixed, one `POST /api/chat/completions` returns `finish_reason=stop` with non-empty content (0.7). | Fresh cluster `k3dgpu-f2-*` (scratch KUBECONFIG, node CPUs 8-15), `nslookup` from a pod before any change, CoreDNS Corefile and node `/etc/resolv.conf` read; root cause fixed in the product if it is a k3d-gpu config gap (regression test first); then one chat through Open WebUI's API with the generated admin account, max_tokens <= 64, temperature 0.6. SUPPORTED: chat returns. REFUTED: no chat returns after the fix. Run (2026-10-17, i9-13900T, k3s v1.35.5, node CPUs 8-15): the prediction's mechanism was wrong. CoreDNS forwarding works (node resolv.conf is k3d's 192.168.32.1 DNAT to Docker's 127.0.0.11; busybox and glibc lookups from pods resolve). The failure is a startup race: `k3d-gpu up` returned on node Ready, when the CoreDNS Deployment was 1 s old (image still pulling); a glibc pod started then got `gaierror(-3, 'Temporary failure in name resolution')` for 11 s (4 tries), first success at 13 s. Fix: `up` waits for the CoreDNS rollout; after it, the same pod resolved on its first try (0.0 s). Then `quickstart --remote https://gaudi.farmgpu.net` with no hostAliases: Open WebUI listed `zai-org/GLM-5.3` and one `POST /api/chat/completions` returned finish_reason=stop, content 'OK', 0.41 s; 0 name resolution errors in its log. | SUPPORTED | @@ -102,9 +102,9 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H7x | (Arms B and C superseded by [amendment V1](#h7x-amendment-v1-eat-release-110-for-arms-b-and-c-2026-10-29): EAT release-1.1.0's stock, vLLM CPU v0.28.0; H7's vLLM version is no longer kept.) H7's hypothesis on the Xeon 6741P, run to H7's full protocol: OVMS's decode lead over EAT's vLLM CPU engine comes from weight precision, not from the engine. One checkpoint (`Qwen/Qwen3-4B` @ `1cfa9a72`), OVMS 2026.4.1 on its 16-bit IR (`OpenVINO/Qwen3-4B-fp16-ov` @ `00298f1c`) and vLLM CPU v0.24.0 in bf16 with EAT's chart defaults decode within 15% of each other, and land within 15% in effective weight bandwidth. A replication: H7's models and versions are kept. | Written 2026-10-20, before any H7x measurement. On AMX both engines compute in bf16 and batch-1 decode reads the same 8.0 GB per token, so the engine gap shrinks: R = OVMS/vLLM decode 1.10 (0.85-1.40); SUPPORTED 0.45, REFUTED 0.30, INCONCLUSIVE 0.25. Reported, not scored: prefill TTFT on ~~\~1936~~ ~1850 tokens (corrected in the 2026-10-21T22:30Z revision to the prompt `h7x.sh` has always sent, 56 copies of one sentence; value unchanged; struck through 2026-10-29) vLLM/OVMS 1.5 (0.7-4; the i9 gap of 4.3x was outside H7's prediction), C/B decode (`--enforce-eager` dropped) 1.0-1.3, decode incremental J/token per arm. | `bench/apb/h7x.sh` (H7's `h7.sh` extended to the pre-registered protocol: H2's workloads and energy method, blocks A B B A A B B A A B, then one C block with `--enforce-eager` dropped), scored by `bench/apb/h7x.py`; decision rule, host changes and hold in [H7x pre-registration](#h7x-pre-registration-h7-to-its-full-protocol-on-the-xeon-6741p-2026-10-20). B and C are EAT release-1.1.0's stock (`0c726f57`, vLLM CPU v0.28.0), after a further setup smoke, per [amendment V1](#h7x-amendment-v1-eat-release-110-for-arms-b-and-c-2026-10-29). | PENDING | | H55 | On the i9-13900T, an SLO gate on the E-core neighbour's `cpu.max`, set from the P-core inference server's own median TTFT, keeps CPU-served prefill within +10% of quiet while the neighbour keeps more throughput than H5's frequency-cap frontier (CMIN: +23.3% at 72.4% neighbour loss) and than the best static quota at the same penalty. | Written 2026-10-20, before any H55 measurement; control rule revised the same day, still before any measurement (see the section). Pre-registered prior, not a result: N0 about +40% (H5 replicate), K50 +20 to +30% at about 50% loss, K25 +8 to +15% at about 75% loss. A_slo climbs only while the penalty at its level is at most +5%, so it is expected to stay at the 25-37.5% levels, close to K25: +8 to +15% at 70-78% loss, failing (b) or (c). Expected verdict REFUTED: package power is the coupling, duty-cycling the neighbour is less power-efficient per op than H5's frequency cap, and a closed-loop c = 1 prefill load leaves no idle gap to admit it into. If H55 holds, A_slo settles at 50% or above at +10% or less. | H5 amendment A1's host-level harness (OVMS 2026.4.1, `OpenVINO/Qwen3-4B-int4-ov`, `bench/prefill-guard/prompt.txt` with a unique tag per request, CPUs 0-14; `stress-ng --cpu 15 --cpu-method matrixprod` on CPUs 17-31; tool proxy on CPU 16), with a 900 s closed-loop prefill window per run (16 tokens, concurrency 1). Arms Q, N0 (`cpu.max max`), K50 (`750000 100000`), K25 (`375000 100000`) and A_slo (`bench/admission/ctl.py` steps 25-100% of 15 CPUs on windows of 20 requests: down if the window median > 1.10 x the round's quiet reference, the median of 20 pre-roll requests; up if it is at most 1.05 x that), 5x5 Latin square, one round per lock hold. Supported only if A_slo's median penalty is at most +10%, its neighbour loss is below 72.4%, and no static arm reaches +10% at a loss no higher than A_slo's plus the neighbour's own N0 run-to-run spread; inconclusive gates and the hold cap in [H55 pre-registration](#h55-pre-registration-slo-gated-admission-of-an-e-core-neighbour-2026-10-20). Not run. A SUPPORTED H55 does not ship a controller by itself: kubelet owns the sandbox pod's `cpu.max` (`limits cpu "1"`). | PENDING | | H56 | Two measurement holds in disjoint cpuset slices of the Xeon 6741P do not perturb each other: an A1 B24 agent block in slice 0 (CPUs 0-23,48-71) behaves equivalently, in per-attempt time, within-budget success and tasks completed, alone and with a concurrent identical A1 hold in slice 1 (CPUs 24-47,72-95), without an all-core clock drop on slice 0 and with the package below its power limit. It decides whether crescent may run two A1-or-lighter holds at once ([adoption rule](#h56-pre-registration-parallel-slice-holds-on-crescent-2026-10-20)); it is not a k3d-gpu product claim. | Written 2026-10-20, before any run; predictions unchanged by the 2026-10-21 revision, priors updated to its narrower bands. Geometric-mean W/A time ratio 1.04 (0.99-1.09); paired success W - A -0.01 (-0.06 to +0.04); completion W/A 0.97 (0.90-1.00); own-slice MHz W/A 0.97; W package 0.75 PL1 (0.55-0.95), A package 0.5 PL1; time pairs 80 (50-96). Batch-1 int4 decode is DRAM-bound, and the sibling's decode and its stream load share DRAM, L3 and package power. Verdict priors: SUPPORTED 0.30, INCONCLUSIVE 0.45, REFUTED 0.25. | Stage 0 first, read-only: `bench/eatflow/crescent/lockwait.py` over both campaigns' `campaign.log`; if lock wait our slots could remove is under 0.10 of hold time, stage 1 is not run and the row stays PENDING (`h56.sh` refuses through `h56.py gate`). [Amendment S0](#h56-pre-registration-parallel-slice-holds-on-crescent-2026-10-20) (2026-10-21, before any stage-1 hold, stage-0 reading disclosed): the share is pooled lock wait over hold time across every campaign log on crescent, nothing netted out; bar 0.10 unchanged. Then one W wiring smoke, which must show slice-0 setup <= 900 s and >= 3 tasks in 120 s. Stage 1: `bench/eatflow/crescent/h56.sh N` for N = 1..8, order A W W A A W W A, one condition-hold per exclusive campaign lock (slot locks first). A: slice 0 alone, slice 1 idle. W: the sibling in slice 1 first (cap 3000 s, budget 1800 s), slice 0 from the sibling's run budget on (cap 2400 s, budget 720 s, the same as A). Hold pairs (1,2) (4,3) (5,6) (8,7), tasks by id; 90% two-stage bootstrap CIs (hold pairs, then tasks; 10,000, seed 56; TOST at alpha 0.05). SUPPORTED: all 8 positions filled, time CI inside [1/1.05, 1.05], success CI inside [-1/24, +1/24], completion W/A >= 1/1.05, >= 60 time pairs, own-slice MHz W/A >= 0.95, W package < 0.95 PL1 and PL1 read. REFUTED: the time CI wholly outside its band, the success CI wholly outside its band, or the completion CI wholly below 1/1.05. Otherwise INCONCLUSIVE. VOID holds are replaced, at most 2. `bench/eatflow/crescent/h56.py score`. Stage 0 under amendment S0 (`lockwait.json`, 62fdf617): pooled share 0.9987 (16081 s of lock wait over 16102 s held, 21 holds, 4 logs), so stage 1 ran; smoke-W passed `h56.py gate` (slice-0 setup 225 s, 15 tasks in 120 s). Stage 1 run 2026-10-21, holds 1-8 in order, every rc 0, none gate-flagged ([H56 run](#h56-run-stage-1-xeon-6741p-2026-10-21)): `h56.py score` reads "4 VOID condition-holds > 2; position 2 missing; position 3 missing; position 6 missing; position 7 missing". Every W hold is VOID on one rule, slice 0's block window not inside the sibling's: a B24 block ends at its 24th task, not at its run budget, and took 178-190 s, so the sibling's block (1800 s budget) ended 27-32 s after slice 0's began and covered 0.15-0.17 of it. No time, success, completion, MHz or power statistic is scored. The 4 A holds are valid (22/24 passed in each). A VOID replacement cannot lift the verdict (4 VOID > 2) and would VOID by the same timing, so no further H56 hold is owed; crescent rows stay K=1 (adoption rule 1). | INCONCLUSIVE | -| H59 | Engine choice by evidence on the Xeon 6741P (crescent; AMX, CPU only, no card): with Gemma 4 26B-A4B QAT at agent concurrency N=4, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter (L, `agent up` with the llama.cpp backend of PR #182) decodes at least 1.2x faster than OVMS int4 (O, `OpenVINO/gemma-4-26b-a4b-it-int4-ov`, the model optimizer's current pick), with GSM8K quality within 2 of 50. SUPPORTED makes llama.cpp the optimizer's CPU engine default for this CPU class (`console/ledger-rules.json`, H59 family); otherwise OVMS stays. | Written 2026-10-21, before any H59 measurement. Informed by H54a's descriptive N=1 numbers on this host (Lg-mtp 54.52, A1g 23.25 tok/s, not a verdict): N=1 L/O 2.2 (1.6-2.8) (0.6); N=4 per-request decode L/O 1.6 (1.1-2.3) (0.55), with MTP's gain smaller at N=4 than at N=1 (0.65); N=16 L/O < N=4 L/O (0.6); prefix-reuse added-agent TTFT O < L (0.55); GSM8K both >= 46/50 (0.7). SUPPORTED (0.55), REFUTED (0.3), INCONCLUSIVE (0.15). | [H59 pre-registration](#h59-pre-registration-engine-choice-per-cpu-class-2026-10-21): arms O and L at their best documented settings, one serving at a time on the CPU, 3 performance holds in rotated order (N=1: 8, N=4: 16, N=16: 32 requests, 256 tokens, greedy), a TTFT and a prefix-reuse phase per arm, 1 quality hold (GSM8K 0-49), RAPL J/token. Statistic: ratio of L's to O's median per-request decode tok/s at N=4 over the pooled holds, 95% bootstrap CI (H54's `boot_ratio`). SUPPORTED: CI lower bound >= 1.2 and L's GSM8K >= O's - 2. REFUTED: CI upper bound < 1.2, or the quality gate fails. Otherwise INCONCLUSIVE (also: an arm not deliverable, load unmatched). Scorer: `bench/h59/h59.py`, [H59 amendment S1](#h59-amendment-s1-the-scorer-and-the-desktop-and-wildct-priors-after-the-h57d-run-2026-10-28) (2026-10-28, before any H59 measurement). O serves base-model int4 (OVB), not QAT weights, so a verdict compares the two stacks as shipped (S1's description fixes); SUPPORTED also needs L faster than O in each performance hold: [H59 amendment S2](#h59-amendment-s2-supported-needs-l-faster-in-each-performance-hold-2026-10-29) (2026-10-29, before any H59 measurement). | PENDING | -| H59d | H59 on the desktop i9-13900T (8 P-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX): same arms, statistic and rule; SUPPORTED makes llama.cpp the CPU engine default for the i9-13900T class only. | Written 2026-10-21, before any H59d measurement; H57d's desktop arms (Od, Ld, Ld-mtp) have no scored result yet. N=1 L/O 1.4 (0.9-2.0) (0.5); N=4 L/O 1.2 (0.8-1.7) (0.45). SUPPORTED (0.35), REFUTED (0.45), INCONCLUSIVE (0.2). | As H59, with the desktop knobs of the H57d pre-registration (P-core balloon, its load gate, H57d's lock and disk rules); N=16 is not run (8 physical P-cores; H57d dropped it for the same reason). Prior restated 2026-10-28 after the H57d run, before any H59d measurement, prediction unchanged; scorer `bench/h59/h59.py --load h57d`: [H59 amendment S1](#h59-amendment-s1-the-scorer-and-the-desktop-and-wildct-priors-after-the-h57d-run-2026-10-28). SUPPORTED also needs L faster than O in each performance hold: [H59 amendment S2](#h59-amendment-s2-supported-needs-l-faster-in-each-performance-hold-2026-10-29) (2026-10-29, before any H59d measurement). | PENDING | -| H59w | H59 on wildct, the client i7-13700 (8 P-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX): same arms, statistic and rule; SUPPORTED makes llama.cpp the CPU engine default for the i7-13700 class only. | Written 2026-10-21, before any H59w measurement. N=1 L/O 1.4 (0.9-2.0) (0.5); N=4 L/O 1.2 (0.8-1.7) (0.45). SUPPORTED (0.35), REFUTED (0.45), INCONCLUSIVE (0.2). | As H59d, with the wildct knobs of the wildct pre-registration (the GPU opt-out, rules W2-W6, hold-start `load1` <= 12). Prior restated 2026-10-28 after the H57d run, before any H59w measurement, prediction unchanged; scorer `bench/h59/h59.py --load h57d`: [H59 amendment S1](#h59-amendment-s1-the-scorer-and-the-desktop-and-wildct-priors-after-the-h57d-run-2026-10-28). SUPPORTED also needs L faster than O in each performance hold: [H59 amendment S2](#h59-amendment-s2-supported-needs-l-faster-in-each-performance-hold-2026-10-29) (2026-10-29, before any H59w measurement). | PENDING | +| H59 | Engine choice by evidence on the Xeon 6741P (crescent; AMX, CPU only, no card): with Gemma 4 26B-A4B QAT at agent concurrency N=4, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter (L, `agent up` with the llama.cpp backend of PR #182) decodes at least 1.2x faster than OVMS int4 (O, `OpenVINO/gemma-4-26b-a4b-it-int4-ov`, the model optimizer's current pick), with GSM8K quality within 2 of 50. SUPPORTED makes llama.cpp the optimizer's CPU engine default for this CPU class (`console/ledger-rules.json`, H59 family); otherwise OVMS stays. | Written 2026-10-21, before any H59 measurement. Informed by H54a's descriptive N=1 numbers on this host (Lg-mtp 54.52, A1g 23.25 tok/s, not a verdict): N=1 L/O 2.2 (1.6-2.8) (0.6); N=4 per-request decode L/O 1.6 (1.1-2.3) (0.55), with MTP's gain smaller at N=4 than at N=1 (0.65); N=16 L/O < N=4 L/O (0.6); prefix-reuse added-agent TTFT O < L (0.55); GSM8K both >= 46/50 (0.7). SUPPORTED (0.55), REFUTED (0.3), INCONCLUSIVE (0.15). | [H59 pre-registration](#h59-pre-registration-engine-choice-per-cpu-class-2026-10-21): arms O and L at their best documented settings, one serving at a time on the CPU, 3 performance holds in rotated order (N=1: 8, N=4: 16, N=16: 32 requests, 256 tokens, greedy), a TTFT and a prefix-reuse phase per arm, 1 quality hold (GSM8K 0-49), RAPL J/token. Statistic: ratio of L's to O's median per-request decode tok/s at N=4 over the pooled holds, 95% bootstrap CI (H54's `boot_ratio`). SUPPORTED: CI lower bound >= 1.2 and L's GSM8K >= O's - 2. REFUTED: CI upper bound < 1.2, or the quality gate fails. Otherwise INCONCLUSIVE (also: an arm not deliverable, load unmatched). Scorer: `bench/h59/h59.py`, [H59 amendment S1](#h59-amendment-s1-the-scorer-and-the-desktop-and-wildct-priors-after-the-h57d-run-2026-10-28) (2026-10-28, before any H59 measurement). O serves base-model int4 (OVB), not QAT weights, so a verdict compares the two stacks as shipped (S1's description fixes); SUPPORTED also needs L faster than O in each performance hold: [H59 amendment S2](#h59-amendment-s2-supported-needs-l-faster-in-each-performance-hold-2026-10-29) (2026-10-29, before any H59 measurement). The runner: [H59 amendment R1](#h59-amendment-r1-the-runner-and-the-host-profiles-2026-10-29) (2026-10-29, before any H59 measurement). | PENDING | +| H59d | H59 on the desktop i9-13900T (8 P-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX): same arms, statistic and rule; SUPPORTED makes llama.cpp the CPU engine default for the i9-13900T class only. | Written 2026-10-21, before any H59d measurement; H57d's desktop arms (Od, Ld, Ld-mtp) have no scored result yet. N=1 L/O 1.4 (0.9-2.0) (0.5); N=4 L/O 1.2 (0.8-1.7) (0.45). SUPPORTED (0.35), REFUTED (0.45), INCONCLUSIVE (0.2). | As H59, with the desktop knobs of the H57d pre-registration (P-core balloon, its load gate, H57d's lock and disk rules); N=16 is not run (8 physical P-cores; H57d dropped it for the same reason). Prior restated 2026-10-28 after the H57d run, before any H59d measurement, prediction unchanged; scorer `bench/h59/h59.py --load h57d`: [H59 amendment S1](#h59-amendment-s1-the-scorer-and-the-desktop-and-wildct-priors-after-the-h57d-run-2026-10-28). SUPPORTED also needs L faster than O in each performance hold: [H59 amendment S2](#h59-amendment-s2-supported-needs-l-faster-in-each-performance-hold-2026-10-29) (2026-10-29, before any H59d measurement). The runner: [H59 amendment R1](#h59-amendment-r1-the-runner-and-the-host-profiles-2026-10-29) (2026-10-29, before any H59d measurement). | PENDING | +| H59w | H59 on wildct, the client i7-13700 (8 P-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX): same arms, statistic and rule; SUPPORTED makes llama.cpp the CPU engine default for the i7-13700 class only. | Written 2026-10-21, before any H59w measurement. N=1 L/O 1.4 (0.9-2.0) (0.5); N=4 L/O 1.2 (0.8-1.7) (0.45). SUPPORTED (0.35), REFUTED (0.45), INCONCLUSIVE (0.2). | As H59d, with the wildct knobs of the wildct pre-registration (the GPU opt-out, rules W2-W6, hold-start `load1` <= 12). Prior restated 2026-10-28 after the H57d run, before any H59w measurement, prediction unchanged; scorer `bench/h59/h59.py --load h57d`: [H59 amendment S1](#h59-amendment-s1-the-scorer-and-the-desktop-and-wildct-priors-after-the-h57d-run-2026-10-28). SUPPORTED also needs L faster than O in each performance hold: [H59 amendment S2](#h59-amendment-s2-supported-needs-l-faster-in-each-performance-hold-2026-10-29) (2026-10-29, before any H59w measurement). The runner: [H59 amendment R1](#h59-amendment-r1-the-runner-and-the-host-profiles-2026-10-29) (2026-10-29, before any H59w measurement). | PENDING | ## Falsification log @@ -6540,6 +6540,32 @@ drafter, records, 30 GB) and the four images it pulled are removed. cpufreq was `bench/prefill-guard/cpufreq.sh snapshot` before the downloads and after hold 2 are byte-identical (and equal H5's campaign snapshot). +### H57d amendment R2: Od's render names the hold's cluster (2026-10-29) + +Recorded 2026-10-29, before any H57wa or H57wm hold (no wildct hold has measured anything). It +governs every later hold of H57d's runner. Mechanics only: no row's arms, models, statistic, gate, +threshold, hold count or prediction changes. + +**What it fixes.** `bench/h57d/arm.sh` renders Od by running `scripts/k3d-gpu agent manifest` +against the hold's own kubeconfig. Since PR #183 (`a7f95676`, 2026-10-21, after H57d's holds), +every kubectl call of `k3d-gpu` names the context `k3d-${K3D_GPU_CLUSTER:-gpu}`. `arm.sh` did not +set `K3D_GPU_CLUSTER`, so the render asked for context `k3d-gpu`, which a hold's kubeconfig does +not hold. Its node queries failed without a word, and Od rendered with no `k3d-gpu.io/cores: +performance` label and a 1-CPU request (H57d's Od: 16). H57dp's two holds ran so; the H57dp run +section above discloses it as a deviation. + +`arm.sh` now passes `K3D_GPU_CLUSTER=${CLUSTER}` on that call. Nothing else of the render +changes. H57w's Od would have rendered the same way, since its holds run this runner. With the fix +it gets the P-core balloon and the 16-CPU request H57d's Od had, which the wildct pre-registration +states for H57w ("Od's 16-CPU request"). + +A regression test, `OdRendersInTheHoldsCluster` in `bench/h57d/test_h57d.py`, runs `arm.sh` for +Od against the fit fixtures. Its kubectl stand-in answers only for the hold's context. The test +reads the P-core label and 16 CPUs in the render; before the fix it read no label and 1 CPU. H59's +runner sets the same variable on both of its arms +([H59 amendment R1](#h59-amendment-r1-the-runner-and-the-host-profiles-2026-10-29)) and also +refuses an arm whose render lacks the balloon. + ### Wildct pre-registration: client Raptor Lake (i7-13700, 2026-10-20) Written 2026-10-20, before any H52w, H18w, H25w, H28w, H34w or H57w measurement on this host. @@ -12746,6 +12772,244 @@ condition. `bench/h59/test_h59.py` adds a pooled CI above 1.2 with one hold at 0 (INCONCLUSIVE), one hold at exactly 1.0 (INCONCLUSIVE) and at 1.05 (SUPPORTED), and a CI below the bar with one slower hold (still REFUTED). +### H59 amendment R1: the runner and the host profiles (2026-10-29) + +Recorded 2026-10-29, before any H59, H59d or H59w measurement. This is the runner amendment the +pre-registration requires before the first H59-family hold. PR #182 (the llama.cpp backend of +`agent up`) is on main, merged 2026-10-29 18:25Z as `b8504e6b`. Mechanics only: no row's arms, +model files, statistic, quality gate, 1.2 bar, load rule, hold count, cap or prediction changes. +No wiring smoke has run. One runs before each row's first hold, scores nothing, and is disclosed +in that row's run section. + +**Files.** `bench/h59/`: + +- `run.py`: the arm table, the hold orders, the host profiles, each arm's render, stage and + check, crescent's load verdict and the prefix phase. +- `hold.sh`: one hold under the host's rules. `hold-run.sh`: the cluster, the P-core sampler and + the arms. `arm.sh`: one arm (render, readiness, records, phases or quality, teardown). +- `prep.sh`: the model files and the drafter, once per host. Also `k3d-shim.sh` and + `reproduce.sh`. +- `bench/wildct/row.sh` gains rows `h59w` and `h59w-prep`. + +These are imported from other rows, not copied: H54's GSM8K items, prompt and grader, its TTFT +prompt, void rule and `boot_ratio`; H57d's P-core sampler and load gate; H25's prefix prompts +(`bench/shared-prefix/probe.py`). Scoring is amendment S1's `bench/h59/h59.py`, unchanged. + +Tests: `bench/h59/test_run.py` and the `h59w` rows of `tests/wildct_hold_test.sh`, in CI. They +render both arms through the real `k3d-gpu agent manifest` against the fit fixtures, with a +kubectl stand-in that answers only for the hold's own context. They run `arm.sh` against a local +OpenAI-compatible server and score its records with `h59.py`. Four holds of a desktop row feed +the bootstrap-ratio scorer end to end. A crescent arm runs its N=16 phase and re-runs a void N=4 +phase. + +**Arms as rendered.** Each arm is `scripts/k3d-gpu agent manifest --model ` at the runner's +commit, run in the hold's cluster. O is `--model google/gemma-4-26B-A4B-it`. L is `--model +google/gemma-4-26B-A4B-it-qat-q4_0-gguf` with `K3D_GPU_AGENT_DRAFT=mtp`. Both set +`K3D_GPU_AGENT_DEVICE=CPU`. The render also sets: + +- `K3D_GPU_CLUSTER` to the hold's cluster. Every node query of `agent manifest` names context + `k3d-${K3D_GPU_CLUSTER:-gpu}`. Without it a hold's own kubeconfig answers "context was not + found", and the arm renders with no P-core label and 1 CPU. That was H57dp's Od (PR #247); see + [H57d amendment R2](#h57d-amendment-r2-ods-render-names-the-holds-cluster-2026-10-29). +- `K3D_GPU_AGENT_REVISION` to the registered commit: O `4e5885c3`, the commit OVB's weights are + fetched at, for the fit's metadata read; L `d1c082be`. An unpinned render reads `@main` + metadata. On the desktop that is a stale cache entry, which the llama.cpp backend refuses + ("metadata names no commit"). For O the launcher also reads the requested base repo's format + at that commit (`agent_model_format`). The base repo has no such commit, so the format reads as + unknown and the render takes OVMS's path, as it does without the pin. +- `K3D_GPU_AGENT_MODEL_NAME=gemma`, the served name. +- `K3D_GPU_TUNE=off`. A host's own tuned profile is not a k3d-gpu default. + +No other agent variable reaches the render. It drops any `K3D_GPU_AGENT_*`, `K3D_GPU_LLAMA_*`, +`K3D_GPU_MODEL_OPT`, `K3D_GPU_MODEL_OPT_LEDGER`, `K3D_GPU_AUTOQUANT`, `K3D_GPU_FIT`, +`K3D_GPU_FIT_RUNG`, `K3D_GPU_FIT_FILE`, `K3D_GPU_CONTEXT`, `K3D_GPU_KUBECONFIG` or +`K3D_GPU_BALLOONS` set in the shell that starts a hold. The variables it passed are recorded +(`render-env.txt`). + +`run.py check` then reads the staged render. A render that fails it stops the arm before +`kubectl apply` (status failed), and the hold counts. It checks: + +- the images: O `openvino/model_server@sha256:a928903f...` (2026.4.1), L + `ghcr.io/ggml-org/llama.cpp:server-b11960@sha256:8052c13e...`; +- device CPU; no GPU resource; +- O's pull container names OVB, CPU, `text_generation` and no draft; +- L's `--model` is the GGUF at `d1c082be` and `--spec-draft-model` the drafter converted from + `95371415`, with `--spec-type draft-mtp`, `--jinja`, `--reasoning off` and numeric `-c`, `-np`, + `--threads` and `--threads-batch`; +- on the desktop and wildct, the P-core label and a 16-CPU request (L: `--threads 8`, + `--threads-batch 16`); on crescent, no core label. + +A later bump of either image pin in the launcher stops the holds until an amendment names it. +Offline renders of this commit, on the desktop fit fixture, give O the fit's `--max_num_seqs=4` +and `--cache_size=20`. They give L `-c 1310720 -np 5 -ngl 0 --threads 8 --threads-batch 16 +--load-mode mmap+mlock`. On a host the fit follows the host's free memory at render time. So each +arm's `fit.json` records what it ran with. `check.json` notes a fit of fewer than 4 sequences, +which makes the N=4 phase queue; that is reported, not gated. + +**Flags read at each pin.** This is the pre-registration's flag check. No flag is missing, so +none is dropped. + +- llama.cpp `b11960`, `common/arg.cpp`: `-m/--model` :3057, `-mu/--model-url` :3066, + `-a/--alias` :3033, `--host` :3343, `--port` :3359, `--jinja` :3727, `-rea/--reasoning` :3746, + `-c` :1650, `-np` :2556, `-ngl` :2811, `-t/--threads` :1528, `-tb/--threads-batch` :1538, + `-lm/--load-mode` :2703 (`mmap+mlock` is one of its values), `--spec-draft-model` :4322 and + `--spec-type` :4330. `draft-mtp` is a type in `common/speculative.cpp` :78. +- Defaults L relies on: the prompt cache is on (`common/common.h` :639, `cache_prompt = true`), + and the draft length is 3 (:331). +- OVMS `v2026.4.1`, `src/cli_parser.cpp`: `--task` :62, `--rest_port` :93, `--log_level` :117, + `--cache_dir` :152, `--pull` :236, `--source_model` :240, `--model_repository_path` :256, + `--model_name` :270, `--model_path` :274 and `--target_device` :314. +- OVMS `src/graph_export/graph_cli_parser.cpp`: `--max_num_seqs` :44, `--enable_prefix_caching` + :52 (default true, so prefix caching is on), `--cache_size` :60, `--reasoning_parser` :84 and + `--tool_parser` :88. This parser allows unrecognised options, so a misspelt pull argument would + be dropped without a word. This read rules that out. + +**Model files.** `prep.sh` runs once per host and is not a hold. + +- Downloads at the registered commits, outside the lock, into `${G}/models`: the GGUF repo's + `gemma-4-26B_q4_0-it.gguf` only, and OVB's snapshot. Each LFS file is checked against the Hub's + sha256 (`bench/wildct/verify_models.py`). +- The GGUF is also reachable as `gguf/google__gemma-4-26B-A4B-it-qat-q4_0-gguf@d1c082be.../`, + the path L's render serves it from. +- OVB's directory and `.ov_cache` are mode 777. OVMS's pull container runs as uid 5000 (H54 + amendment R2's directory mode). +- The drafter is #182's own conversion. `prep.sh` renders L with no cluster and takes the + conversion Job from that render: `full-b11960@sha256:7afc81bf...`, the pins + `transformers==5.21.0 huggingface-hub==2.3.0 tokenizers==0.23.3 safetensors==0.8.0`, + `convert_hf_to_gguf.py --outtype bf16`, then `llama-quantize` Q4_0. The offline render names + it Job `gguf-convert-b459c9c78d78`. `prep.sh` runs that container once with `docker run`, under + the host lock and `timeout -k 30 3600`. It keeps the Job's image, command, arguments, + environment and memory limit, mounts the models at `/models`, and runs as the campaign user, + not root. +- The Job's spec is kept as `prep-job.json`. A hold whose render carries another conversion Job + does not start the arm. +- The drafter's sha256 is recorded in each Reproduce header. It is a different build from H57d's + (`8eb21d5b...`, b11749, transformers 5.20.2), as amendment S1 notes that L already differs from + H57d's Ld-mtp. `${G}/weights.sha256` holds every file the arms read. + +**Staging.** `run.py stage` changes where an arm reads its files, nothing it serves. The k3d shim +binds `${G}/models` into the node at `/var/apb-models`, read-write, because O's pull container +writes `graph.pbtxt`. Weights are never copied into the node. + +- O: `bench/apb/stage.py ovms`, so the PVC becomes that hostPath. The pull container is kept. It + rewrites `graph.pbtxt` from the hold's own arguments and finds OVB's files with no `.git`, so it + downloads nothing. OVMS's compiled-model cache is emptied after each O arm (H54's teardown), so + no boot reads an earlier hold's cache. +- L: the PVC becomes the same hostPath, mounted read-only. The conversion ConfigMap and Job and + the `wait-gguf` init container are dropped; after `prep.sh` each would return at once. So is + `--model-url` with its value: it is llama-server's download source, and the GGUF is already at + `--model`. +- Image, every other argument and its order, requests, limits and labels are as rendered. The + tests compare the staged render with the raw one. + +**Holds.** `hold.sh HOST HOLD [REPEATS]`: + +- Holds 1-3 are performance holds in the orders (O, L), (L, O), (O, L). Hold 4 is the quality + hold, in (L, O). +- A spare 5 or 6 repeats hold REPEATS in its order. It may only repeat a hold that ran and left + an arm not measured. It writes the file `repeats`, which `h59.py` reads, only once it runs. +- A hold runs once. A hold that recorded its `rc` is refused, so a failed hold counts and only a + spare repeats it. There is no hold number above 6. +- Before the lock, `hold.sh` refuses a host whose CPU model is not the row's, a node image not + pinned by digest, a `${G}/bin/k3d` that is not the shim, and missing model files. +- The node image (`H59_IMAGE`) is the k3d-gpu release whose launcher is this commit's or later, + by digest. The row's first hold records it, and later holds must run the same image. +- A wiring smoke (`hold.sh HOST smoke`) runs hold 1's arms with one 16-token request and one + `chat_template_kwargs` chat each. It scores nothing and is disclosed. + +**Host profiles** (`run.py profile`): + +| | crescent (H59) | desktop (H59d) | wildct (H59w) | +|---|---|---|---| +| CPU checked | Xeon 6741P | i9-13900T | i7-13700 | +| Lock | `~/k3dgpu-bench.lock` | the desktop's (`LOCK`) | W1's (outer), then a private inner lock | +| Hold start | `load1` <= 12, no aiperf process (H50) | `load1` <= 24 | `load1` <= 12 (W1, and again inside) | +| Floors | `/` at most 97% used | 15 GB free on `/`, MemAvailable >= 20 GiB | as the desktop, inside W6's 15 + 40 GB | +| Cap, arms end | 2400 s, 2370 s (H54) | 4500 s, 4470 s (H57d) | 4500 s, 4470 s (H57w), inside W1's 4600 s | +| Cluster | `k3dgpu-g-h59-`, `amx`, C1 | `k3dgpu-d-h59-`, `hybrid`, C1 | `k3dgpu-w-h59-`, `auto,-nvidia,-intel`, W3 | +| CPUs | no balloon (half the node) | P-core balloon, 16 CPUs | as the desktop | +| Phases | N=1, N=4, N=16, TTFT, prefix | N=1, N=4, TTFT, prefix | as the desktop | +| Load rule | H54's void rule, `run.py gate` | H57d's gate, `h57d.py sample` | as the desktop | + +wildct runs each hold as `bench/wildct/hold.sh H59w- 4600 40 bench/wildct/row.sh h59w +[]`. That gives it W1's lock, the GPU opt-out, W3, W4's RAPL reader, W5's sampler and W6's +15 + 40 GB start floor. `prep.sh` runs as `row.sh h59w-prep`, outside a hold. + +**Phases.** As registered, in `arm.sh`: + +- the 60 s idle window (RAPL), then 3 warmup requests; +- N=1 (8 requests), N=4 (16 at concurrency 4), N=16 (32 at 16, crescent only), and TTFT (4 at + N=1 on H54's 1,000-token prompt, `max_tokens` 16). Each runs `bench/llm_load.py + --unique-prompts`, `max_tokens` 256, `ignore_eos`, temperature 0, under `bench/energy.sh`; +- prefix reuse: H25's phase S without its control agent (`probe.py` `gen` and `request`). Agents + a1-a6 run in sequence. Each sends the byte-identical system text and 12 tool schemas plus its + own 96-token message, `max_tokens` 8, thinking off, streamed. The seed is 5900 plus the hold's + number. The score is the median first-chunk TTFT of agents 2-6 (`prefix.json`). It records + `usage.prompt_tokens`, which Gemma's tokenizer puts near H25's 1,536, not at it; +- quality: H54's GSM8K 0-49 runner. + +An arm is measured when its N=4 phase completed a request. Otherwise it is failed: the server +stopped answering after readiness (H54 amendment R3). In the quality hold an arm is measured when +GSM8K recorded an item that did not error. The records use H57d's layout, which `h59.py` reads. +Each arm adds `cpus.txt` (the server's command line, cgroup CPU set, `Cpus_allowed_list`, ISA and +threads' affinity), `fit.json`, `check.json`, `deploy.yaml` and `agent.yaml`, and O adds the +`graph.pbtxt` it served. + +**Load.** On the desktop and wildct, H57d's sampler runs beside each hold, and the row is scored +with `h59.py score RESULTS --load h57d`. On crescent, H54's rule (amendment R4) runs in the hold. +Each phase is checked against the serving samples of the arm's idle window, and a void phase is +re-run once after a fresh idle window. `run.py gate RESULTS` writes the verdict that `h59.py +score RESULTS --load GATE.json` reads. The load is unmatched when an N=4 phase of a performance +hold (a spare in place of the hold it repeats) was void on both tries or carries no check. + +**The engine rule as merged.** #182 merged as `b8504e6b`. There `console/model_opt.py optimize +--engine-from-ledger` is the rule amendment S1 describes at #182's head `62ac8749`: + +- With `K3D_GPU_AGENT_BACKEND` unset and no variant named, an H59-family row SUPPORTED for the + host's CPU class makes llama.cpp the CPU engine, when llama.cpp has an int4 GGUF of the model. + That arm drafts with the QAT MTP assistant. +- A REFUTED row for the class keeps OVMS and wins over a SUPPORTED one. PENDING and INCONCLUSIVE + rows pick nothing. +- `agent up` keeps OVMS when `K3D_GPU_AGENT_DRAFT` names an OVMS draft device. + +**Reported for the owner, not acted on: what the desktop load gate reads.** H59d's and H59w's +gate (H57d's, amendment S1) compares the arms' mean foreign P-core load over the N=4 phases. That is the +busy time of CPUs 0-15 minus the k3d node's own CPU, from H57d's sampler. On the desktop's records +(`bench/h57d/results/2026-10-21` and `2026-10-29`) this reading moves with the arm, not with the +host. Foreign P-core load in CPU-s/s, per hold, +over each arm's 60 s idle window (server up, no request) and over its N=4 phase: + +| Records | Ld idle, N=4 | Ld-mtp idle, N=4 | Ld-mtp-u idle, N=4 | Od idle, N=4 | +|---|---|---|---|---| +| H57d hold 1 | 7.20, 8.21 | 9.32, 6.60 | | 6.40, 6.48 | +| H57d hold 2 | 6.25, 6.28 | 6.79, 6.95 | | 6.06, 1.89 | +| H57d hold 3 | 3.06, 2.52 | 3.47, 3.18 | | 2.74, 0.53 | +| H57dp hold 1 | | 3.19, 2.58 | 3.07, 3.37 | 3.67, 0.37 | +| H57dp hold 2 | | 3.87, 3.50 | 3.58, 3.57 | 3.10, 1.17 | + +- In their idle windows the arms read alike. Outside H57d's busiest hold 1, the llama.cpp arms + keep about their idle reading under load, pinned (Ld, Ld-mtp) or not (Ld-mtp-u), and Od's + falls. +- Where Od's reading falls, its own node CPU over N=4 (7.7-8.0 CPU-s/s) is close to the whole + P-core busy time (8.3-9.6). So during Od's phases the host's other processes ran mostly off the + P-cores. During the llama.cpp arms' phases, which decode on 8 threads (`-t 8`), they kept their + share of the P-cores. Why the scheduler places them so (for instance, on idle hyper-thread + siblings beside one-thread-per-core decoding) is not measured here. +- So the gate, read over the phases, compares a quantity the arm itself sets. That is the trap H54 + amendment R4 named for crescent's void rule: a post-treatment variable. +- H59's L runs #182's `--threads 8 --threads-batch 16`, with no pinning, and decodes on 8 threads. + The same pattern is expected. An INCONCLUSIVE (load unmatched) H59d or H59w would then say + little about the host's load. +- Read over the arms' idle windows instead, the arms match. H57d's three arms read 5.50, 6.53 and + 5.06 (spread 1.46 <= 2.00); H59's pair on those records, Ld-mtp and Od, reads 6.53 and 5.06. + H57dp's three arms read 3.53, 3.33 and 3.38 (spread 0.21). Over the N=4 phases H57d's pair reads + 5.58 and 2.96, spread 2.61 > 2.00 (amendment S1's table). + +Changing H59d's and H59w's gate, for instance to read the idle windows as H54 amendment R4 did, is +a protocol change and the owner's decision. It would need a dated amendment before the first H59d +or H59w hold. This amendment does not make it. The runner records both readings: the idle window +is a phase in each arm's `phases.jsonl`, beside the sampler's `load.jsonl`. + ## Scope corrections Limits on what a SUPPORTED verdict may be quoted as saying. diff --git a/bench/h57d/arm.sh b/bench/h57d/arm.sh index 6f16cf1c..5bdfbdd8 100644 --- a/bench/h57d/arm.sh +++ b/bench/h57d/arm.sh @@ -49,7 +49,11 @@ stamp "start ${MODE} (${left} s of the hold left)" M=${OUT}/deploy.yaml case ${DEPLOY} in k3d-gpu) - K3D_GPU_AGENT_DEVICE=CPU K3D_GPU_AGENT_MODEL=$(field model) K3D_GPU_AGENT_MODEL_NAME=${MODEL} K3D_GPU_AGENT_DRAFT=off \ + # K3D_GPU_CLUSTER: `agent manifest` reads the node's hybrid label and allocatable CPU + # through context k3d-${K3D_GPU_CLUSTER}; unset, that is k3d-gpu, which the hold's + # kubeconfig does not hold, and Od renders with no P-core label and 1 CPU (H57d + # amendment R2). + K3D_GPU_CLUSTER=${CLUSTER} K3D_GPU_AGENT_DEVICE=CPU K3D_GPU_AGENT_MODEL=$(field model) K3D_GPU_AGENT_MODEL_NAME=${MODEL} K3D_GPU_AGENT_DRAFT=off \ K3D_GPU_AUTOQUANT=off "${REPO}/scripts/k3d-gpu" agent manifest > "${OUT}/agent.yaml" 2> "${OUT}/agent.err" \ && ${PY} "${REPO}/bench/apb/stage.py" ovms "${OUT}/agent.yaml" "${OUT}/staged.yaml" \ && ${PY} "${D}/h57d.py" od-ro "${OUT}/staged.yaml" "${M}" ;; diff --git a/bench/h57d/test_h57d.py b/bench/h57d/test_h57d.py index ee4dd662..96e091c9 100644 --- a/bench/h57d/test_h57d.py +++ b/bench/h57d/test_h57d.py @@ -9,6 +9,7 @@ import subprocess import sys import tempfile +import time import unittest HERE = os.path.dirname(os.path.abspath(__file__)) @@ -644,5 +645,74 @@ def test_engine_flags_block(self): self.assertIn(" llamacpp: " + h57d.LLAMACPP, lines) +ROOT = os.path.abspath(os.path.join(HERE, "..", "..")) +FIX = os.path.join(ROOT, "tests", "fixtures", "fit") +# kubectl stand-in: `k3d-gpu` names --context on every call; a hold's kubeconfig holds only the +# hold's cluster, so any other context reads "context was not found". apply fails, which stops +# arm.sh right after the render. +KUBECTL = r'''#!/usr/bin/env python3 +import os, sys +a = sys.argv[1:] +ctx = None +while a and a[0] == "--context": + ctx, a = a[1], a[2:] +if ctx is not None and ctx != "k3d-" + os.environ["STUB_CLUSTER"]: + sys.exit(f"error: context was not found for specified context: {ctx}") +s = " ".join(a) +if s == "get nodes -l k3d-gpu.io/hybrid-cpu=true -o name": + print("node/k3d-x-server-0") +elif "allocatable" in s: + print("cpu 32") +elif a[:1] == ["apply"]: + sys.exit("apply refused by the test") +''' + + +@unittest.skipUnless(HAVE_YAML, "PyYAML") +class OdRendersInTheHoldsCluster(unittest.TestCase): + """H57d amendment R2: arm.sh renders Od with K3D_GPU_CLUSTER set to the hold's cluster, so + `agent manifest` reads the hold's hybrid node: the P-core label and the 16 P-core CPUs (H57dp's + deviation was a render through context k3d-gpu: no label, 1 CPU).""" + + def test_od_render_has_the_pcore_balloon(self): + with tempfile.TemporaryDirectory() as d: + b = os.path.join(d, "bin") + os.makedirs(b) + with open(os.path.join(b, "kubectl"), "w") as f: + f.write(KUBECTL) + os.chmod(os.path.join(b, "kubectl"), 0o755) + cache = os.path.join(d, "cache") + os.makedirs(cache) + for n in os.listdir(os.path.join(FIX, "models")): + src = os.path.join(FIX, "models", n) + os.symlink(src, os.path.join(cache, f"{n}@main")) + sysd = os.path.join(d, "root", "sys", "devices") + for name, cpus in (("cpu_core", "0-15"), ("cpu_atom", "16-31")): + os.makedirs(os.path.join(sysd, name)) + with open(os.path.join(sysd, name, "cpus"), "w") as f: + f.write(cpus + "\n") + open(os.path.join(d, "kubeconfig"), "w").close() + env = {"PATH": b + ":/usr/bin:/bin", "HOME": d, "REPO": ROOT, "CLUSTER": "k3dgpu-d-h57d-1", + "STUB_CLUSTER": "k3dgpu-d-h57d-1", "KUBECONFIG": os.path.join(d, "kubeconfig"), + "DEADLINE": str(int(time.time()) + 5000), "MODE": "perf", + "K3D_GPU_SYSROOT": os.path.join(d, "root"), "K3D_GPU_FIT_CACHE": cache, + "K3D_GPU_FIT_HOST": os.path.join(FIX, "hosts", "desktop-i9-nogpu.json"), + "K3D_GPU_HF_FIXTURES": os.path.join(ROOT, "tests", "fixtures", "hf"), + "XDG_STATE_HOME": os.path.join(d, "state"), "XDG_CACHE_HOME": os.path.join(d, "xdg")} + out = os.path.join(d, "Od") + r = subprocess.run(["bash", os.path.join(HERE, "arm.sh"), "Od", out], env=env, + capture_output=True, text=True, timeout=300) + with open(os.path.join(out, "status.json")) as f: + self.assertIn("apply failed", json.load(f)["reason"], r.stdout + r.stderr) + with open(os.path.join(out, "agent.err")) as f: + err = f.read() + with open(os.path.join(out, "deploy.yaml")) as f: + docs = [x for x in yaml.safe_load_all(f) if x] + dep = [x for x in docs if x["kind"] == "Deployment"][0] + tpl = dep["spec"]["template"] + self.assertEqual(tpl["metadata"]["labels"].get("k3d-gpu.io/cores"), "performance", err) + self.assertEqual(str(tpl["spec"]["containers"][0]["resources"]["requests"]["cpu"]), "16", err) + + if __name__ == "__main__": unittest.main() diff --git a/bench/h59/arm.sh b/bench/h59/arm.sh new file mode 100644 index 00000000..d993a779 --- /dev/null +++ b/bench/h59/arm.sh @@ -0,0 +1,203 @@ +#!/usr/bin/env bash +# arm.sh ARM OUT: one H59-family arm in the hold's cluster (EXPERIMENTS.md "H59 pre-registration" +# and "H59 amendment R1"): render it with `k3d-gpu agent manifest --model` in the hold's cluster, +# stage and check it (run.py render), wait for the model list and a 16-token chat, record the +# server's image, command line, CPUs and fit, run the hold's phases (MODE=perf: 60 s idle, 3 +# warmup requests, N=1, N=4, N=16 on crescent, TTFT, prefix reuse; MODE=quality: GSM8K items +# 0-49), then delete the arm. +# REPO this repository; G the campaign dir (models under G/models) +# CLUSTER the hold's cluster; KUBECONFIG its scratch kubeconfig; H59_PROFILE the host +# DEADLINE epoch second the hold's arms must end by; MODE perf | quality +# HOLDNUM the hold's number (the prefix phase's seed); SERVING crescent's serving.jsonl +# H59_PORT local port (default random); H59_IDLE_S idle window (default 60) +# status.json: measured | not_deliverable | not_reached | failed | smoke, with the reason. +set -uo pipefail +ARM=$1 OUT=$2 +mkdir -p "${OUT}" +D=${REPO:?}/bench/h59 +H=${REPO}/bench/h54 +PY=python3 +RUN=${D}/run.py +LOAD=${REPO}/bench/llm_load.py +ENERGY=${REPO}/bench/energy.sh +BOOT=900 +IDLE=${H59_IDLE_S:-60} +now() { date +%s; } +stamp() { echo "$(date -u +%FT%TZ) ${ARM} $*" | tee -a "${OUT}/arm.log"; } +status() { ${PY} -c 'import json,sys; json.dump({"status": sys.argv[1], "reason": sys.argv[2], "at": sys.argv[3]}, open(sys.argv[4], "w"))' \ + "$1" "$2" "$(date -u +%FT%TZ)" "${OUT}/status.json"; stamp "status $1: $2"; } +field() { ${PY} "${RUN}" field "${ARM}" "$1"; } +pfield() { ${PY} "${RUN}" profile "${H59_PROFILE:?}" | sed -n "s/^$1=//p"; } +NODE=k3d-${CLUSTER:?}-server-0 +NSP=k3d-gpu-agent SVC=$(field svc) SPORT=$(field port) API=$(field api) +PHASES=$(pfield phases) RULE=$(pfield load_rule) +MODEL=gemma +PORT=${H59_PORT:-$(( 19000 + RANDOM % 1000 ))} +pf="" +pf_loop() { # kubectl port-forward can drop on a long phase: respawn it + trap 'kill "${k}" 2>/dev/null; exit 0' TERM + while :; do + kubectl -n "${NSP}" port-forward "svc/${SVC}" "${PORT}:${SPORT}" >> "${OUT}/pf.log" 2>&1 & + k=$! + wait "${k}" + echo "port-forward exited $(date -u +%FT%TZ)" >> "${OUT}/pf.log" + sleep 1 + done +} +cleanup_pf() { [ -z "${pf}" ] || kill "${pf}" 2>/dev/null; pf=""; } +trap cleanup_pf EXIT + +need=900 +[ "${MODE:?}" = quality ] && need=600 +left=$(( DEADLINE - $(now) )) +if [ "${left}" -lt "${need}" ]; then status not_reached "${left} s of the hold left (< ${need})"; exit 0; fi +stamp "start ${MODE} (${left} s of the hold left)" + +# ---- render, stage, check (run.py render): agent.yaml, deploy.yaml, check.json, fit.json -------- +M=${OUT}/deploy.yaml +if ! why=$(${PY} "${RUN}" render "${ARM}" "${OUT}" 2>&1); then + status failed "$(tail -1 <<< "${why}")" + exit 1 +fi +kubectl apply -f "${M}" > "${OUT}/apply.log" 2>&1 || { status failed "apply failed: $(tail -1 "${OUT}/apply.log")"; exit 1; } + +# ---- readiness: the model list and a 16-token chat ------------------------------------------ +t_apply=$(now) +limit=$(( BOOT < DEADLINE - t_apply - 300 ? BOOT : DEADLINE - t_apply - 300 )) +base="http://127.0.0.1:${PORT}${API}" +ready="" +while [ $(( $(now) - t_apply )) -lt "${limit}" ]; do + if [ -z "${pf}" ] || ! kill -0 "${pf}" 2>/dev/null; then + pf="" + if kubectl -n "${NSP}" get pods -l "app=${SVC}" -o jsonpath='{.items[*].status.phase}' 2>/dev/null | grep -q Running; then + pf_loop & + pf=$! + sleep 3 + fi + fi + if [ -n "${pf}" ] && curl -fsS --max-time 10 "${base}/models" > "${OUT}/models.json" 2>/dev/null \ + && curl -fsS --max-time 120 "${base}/chat/completions" -H 'Content-Type: application/json' \ + -d "{\"model\":\"${MODEL}\",\"max_tokens\":16,\"messages\":[{\"role\":\"user\",\"content\":\"Say hello.\"}]}" \ + > "${OUT}/hello.json" 2>/dev/null; then + ready=1 + break + fi + sleep 10 +done +t_ready=$(now) +echo "{\"apply\": ${t_apply}, \"ready\": ${t_ready}, \"boot_s\": $(( t_ready - t_apply )), \"limit_s\": ${limit}}" > "${OUT}/boot.json" +kubectl get pods -A -o wide > "${OUT}/pods.txt" 2>&1 +pod=$(kubectl -n "${NSP}" get pods -l "app=${SVC}" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) +ctr=$(kubectl -n "${NSP}" get pod "${pod}" -o jsonpath='{.spec.containers[0].name}' 2>/dev/null) +# cpu_usec: the k3d node container's cgroup CPU time, or nothing when it cannot be read. +cpu_usec() { + local id + id=$(docker inspect -f '{{.Id}}' "${NODE}" 2>/dev/null) || return 1 + awk '/^usage_usec/{print $2}' "/sys/fs/cgroup/system.slice/docker-${id}.scope/cpu.stat" 2>/dev/null +} +teardown() { + cleanup_pf + kubectl -n "${NSP}" logs "${pod}" --all-containers > "${OUT}/server.log" 2>&1 + kubectl -n "${NSP}" describe pod "${pod}" > "${OUT}/describe.txt" 2>&1 + kubectl delete -f "${M}" --wait=true --timeout=300s > "${OUT}/delete.log" 2>&1 + # OVMS's compiled-model cache leaves with the arm, so no later boot reads it (H54's teardown). + [ "${ARM}" != O ] || docker exec "${NODE}" sh -c 'rm -rf /var/apb-models/.ov_cache/*' >> "${OUT}/delete.log" 2>&1 + # The node's CPU reads idle (< 1 CPU-s/s over 10 s) before the next arm, at most 180 s; a + # cgroup that cannot be read ends the wait (recorded). + local a b + for _ in $(seq 18); do + a=$(cpu_usec) && [ -n "${a}" ] || { echo "idle wait: node cgroup not readable" >> "${OUT}/delete.log"; break; } + sleep 10 + b=$(cpu_usec) && [ -n "${b}" ] || break + [ $(( (b - a) / 10000000 )) -lt 1 ] && break + done +} +if [ -z "${ready}" ]; then + if [ "${limit}" -lt "${BOOT}" ]; then status not_reached "not ready within the ${limit} s the hold left (boot limit ${BOOT} s)" + else status not_deliverable "no model list and 16-token chat within ${BOOT} s"; fi + teardown + exit 0 +fi +stamp "ready after $(( t_ready - t_apply )) s" +kubectl -n "${NSP}" get pod "${pod}" -o jsonpath='{.spec.containers[0].image} {.status.containerStatuses[0].imageID}' > "${OUT}/image.txt" +kubectl -n "${NSP}" get pod "${pod}" -o json > "${OUT}/pod.json" +# The server's command line, the CPU set it got (the P-core balloon on a hybrid host), the ISA it +# sees and its threads' affinity. +kubectl -n "${NSP}" exec "${pod}" -c "${ctr}" -- sh -c \ + 'echo "cmdline $(tr "\0" " " < /proc/1/cmdline)"; echo "cpuset $(cat /sys/fs/cgroup/cpuset.cpus.effective)"; grep Cpus_allowed_list /proc/1/status; echo "avx512f $(grep -c avx512f /proc/cpuinfo) amx $(grep -c amx_tile /proc/cpuinfo) avx_vnni $(grep -c avx_vnni /proc/cpuinfo) nproc $(nproc)"; echo "server threads by affinity:"; for t in /proc/1/task/*; do grep -s Cpus_allowed_list "${t}/status" | cut -f2; done | sort | uniq -c' > "${OUT}/cpus.txt" 2>&1 +# O: the graph.pbtxt the --pull init container wrote from this hold's arguments. +[ "${ARM}" != O ] || kubectl -n "${NSP}" exec "${pod}" -c "${ctr}" -- cat "/models/OpenVINO/gemma-4-26b-a4b-it-int4-ov/graph.pbtxt" > "${OUT}/graph.pbtxt" 2>&1 + +if [ -n "${SMOKE:-}" ]; then + # Wiring smoke (discarded, disclosed): one streamed 16-token request with ignore_eos, one chat + # with chat_template_kwargs and the prefix phase's first agent, then the arm is deleted. + ${PY} "${LOAD}" --base-url "${base}" --model "${MODEL}" --requests 1 --concurrency 1 --warmup 0 --max-tokens 16 \ + --ignore-eos --unique-prompts --out "${OUT}/smoke-load.json" >> "${OUT}/arm.log" 2>&1 + curl -sS --max-time 300 "${base}/chat/completions" -H 'Content-Type: application/json' \ + -d "{\"model\":\"${MODEL}\",\"max_tokens\":16,\"chat_template_kwargs\":{\"enable_thinking\":false},\"messages\":[{\"role\":\"user\",\"content\":\"Say hello.\"}]}" \ + > "${OUT}/smoke-kwargs.json" 2>&1 + status smoke "ready; wiring checks in smoke-*.json" + teardown + exit 0 +fi + +# ---- phases ---------------------------------------------------------------------------------- +W0="" W1="" +idle() { # idle [SUFFIX]: the 60 s idle window (RAPL); on crescent also the void rule's CPU window + local t0 + t0=$(date +%s.%N) + bash "${ENERGY}" --zone intel-rapl:0 --interval 5 --json "${OUT}/energy-idle${1:+-$1}.json" -- sleep "${IDLE}" 2>> "${OUT}/arm.log" + W0=${t0} W1=$(date +%s.%N) + echo "{\"phase\": \"idle${1:+-$1}\", \"t0\": ${W0}, \"t1\": ${W1}}" >> "${OUT}/phases.jsonl" +} +load() { # load NAME REQUESTS CONCURRENCY MAX_TOKENS [extra llm_load args] + local name=$1 n=$2 c=$3 mt=$4 + shift 4 + bash "${ENERGY}" --zone intel-rapl:0 --interval 5 --json "${OUT}/energy-${name}.json" -- \ + ${PY} "${LOAD}" --base-url "${base}" --model "${MODEL}" --requests "${n}" --concurrency "${c}" --warmup 0 \ + --max-tokens "${mt}" --temperature 0 --unique-prompts --timeout 1800 --label "H59 ${ARM} ${name}" \ + --out "${OUT}/load-${name}.json" "$@" 2>> "${OUT}/arm.log" +} +phase() { # phase NAME REQUESTS CONCURRENCY MAX_TOKENS [extra]: run and record; on crescent (H54's + # rule, amendment R4) a void phase is re-run once after a fresh idle window + local name=$1 t0 t1 v try + for try in 1 2; do + t0=$(date +%s.%N) + load "$@" + t1=$(date +%s.%N) + if [ "${RULE}" != h54-void ]; then + echo "{\"phase\": \"${name}\", \"t0\": ${t0}, \"t1\": ${t1}}" >> "${OUT}/phases.jsonl" + return + fi + v=$(${PY} "${H}/h54.py" void "${SERVING:?}" "${t0}" "${t1}" "${W0}" "${W1}") + echo "{\"phase\": \"${name}\", \"try\": ${try}, \"t0\": ${t0}, \"t1\": ${t1}, \"w0\": ${W0}, \"w1\": ${W1}, \"check\": ${v}}" >> "${OUT}/phases.jsonl" + case ${v} in *'"void": true'*) [ "${try}" = 1 ] && { mv "${OUT}/load-${name}.json" "${OUT}/load-${name}-void1.json"; idle "${name}"; continue; } ;; esac + return + done +} +has() { [[ ",${PHASES}," == *",$1,"* ]]; } +if [ "${MODE}" = perf ]; then + idle + load warmup 3 1 256 --ignore-eos + phase n1 8 1 256 --ignore-eos + phase n4 16 4 256 --ignore-eos + if has n16; then phase n16 32 16 256 --ignore-eos; fi + phase ttft 4 1 16 --prompt "$(cat "${H}/ttft-prompt.txt")" + t0=$(date +%s.%N) + ${PY} "${RUN}" prefix "${base}" "${MODEL}" "$(( 5900 + ${HOLDNUM:-0} ))" "${OUT}/prefix.json" >> "${OUT}/arm.log" 2>&1 + echo "{\"phase\": \"prefix\", \"t0\": ${t0}, \"t1\": $(date +%s.%N)}" >> "${OUT}/phases.jsonl" + # Measured when the N=4 phase completed a request (h59.py scores the completed ones); a server + # that stopped answering after readiness is failed (H54 amendment R3). + if ${PY} -c 'import json,sys; s=json.load(open(sys.argv[1]))["phases"][0]["samples"]; sys.exit(not any(x.get("ok") for x in s))' \ + "${OUT}/load-n4.json" 2>/dev/null; then status measured "perf phases recorded" + else status failed "server stopped answering: the N=4 phase completed no request"; fi +else + t0=$(date +%s.%N) + ${PY} "${H}/h54.py" gsm8k "${base}" "${MODEL}" "${H}/gsm8k-test-0-199.jsonl" "${OUT}/quality.json" 0-49 >> "${OUT}/arm.log" 2>&1 + echo "{\"phase\": \"quality\", \"t0\": ${t0}, \"t1\": $(date +%s.%N)}" >> "${OUT}/phases.jsonl" + if ${PY} -c 'import json,sys; q=json.load(open(sys.argv[1])); sys.exit(not (q.get("n") and q.get("errors", 0) < q["n"]))' \ + "${OUT}/quality.json" 2>/dev/null; then status measured "quality recorded" + else status failed "quality not recorded, or every item errored"; fi +fi +teardown +exit 0 diff --git a/bench/h59/hold-run.sh b/bench/h59/hold-run.sh new file mode 100644 index 00000000..4db2bb16 --- /dev/null +++ b/bench/h59/hold-run.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# hold-run.sh R ARM...: inside hold.sh's lock and timeout: create the hold's cluster with the +# host profile's vendors, node image and kubelet thresholds, start the P-core load sampler on a +# hybrid host (H57d's, which the desktop and wildct load gate reads), run each arm (arm.sh) until +# the profile's arms_end_s after the hold started. CLUSTER, KUBECONFIG, REPO, G, MODE, +# H59_PROFILE, H59_IMAGE (and SERVING on crescent) come from hold.sh; hold.sh deletes the cluster. +set -uo pipefail +R=$1 +shift +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +t0=$(date +%s) +pfield() { python3 "${HERE}/run.py" profile "${H59_PROFILE:?}" | sed -n "s/^$1=//p"; } +# The k3d shim in ${G}/bin adds the campaign's model directory at /var/apb-models in the node. +K3D_GPU_VENDORS=$(pfield vendors) K3D_GPU_IMAGE=${H59_IMAGE:?} K3D_GPU_K3S_ARGS=$(pfield k3s_args) \ + K3D_GPU_CLUSTER=${CLUSTER:?} timeout 300 "${REPO:?}/scripts/k3d-gpu" up > "${R}/up.log" 2>&1 \ + || { echo "up failed"; tail -20 "${R}/up.log"; exit 1; } +kubectl get node -o json > "${R}/node.json" +docker exec "k3d-${CLUSTER}-server-0" ls /var/apb-models > /dev/null || { echo "models volume missing in the node"; exit 1; } +if [ "$(pfield sampler)" = pcore ]; then + python3 "${REPO}/bench/h57d/h57d.py" sample "${R}/load.jsonl" "k3d-${CLUSTER}-server-0" & + sp=$! + trap 'kill "${sp}" 2>/dev/null' EXIT +fi +export DEADLINE=$(( t0 + $(pfield arms_end_s) )) +for arm in "$@"; do + bash "${HERE}/arm.sh" "${arm}" "${R}/${arm}" +done diff --git a/bench/h59/hold.sh b/bench/h59/hold.sh new file mode 100644 index 00000000..6767ce3a --- /dev/null +++ b/bench/h59/hold.sh @@ -0,0 +1,121 @@ +#!/usr/bin/env bash +# hold.sh HOST HOLD [REPEATS]: one H59-family hold (EXPERIMENTS.md "H59 pre-registration" and "H59 +# amendment R1") on HOST = crescent (H59, Xeon 6741P), desktop (H59d, i9-13900T) or wildct (H59w, +# i7-13700, run as bench/wildct/hold.sh's command: bench/wildct/row.sh h59w). HOLD: 1-3 perf in +# the orders (O, L), (L, O), (O, L); 4 quality in (L, O); a spare 5 or 6 repeats hold REPEATS +# (one that ran and left an arm not measured) in its order; smoke runs hold 1's arms with no +# scored request (discarded, disclosed). Each hold runs once: a failed hold counts. +# +# Before the lock it refuses (exit 2) a host whose CPU is not the profile's, a node image not +# pinned by digest or not the one the row's first hold ran, a ${G}/bin/k3d that is not +# bench/h59/k3d-shim.sh, and models prep.sh has not left in ${G}/models. Then the host's own rules +# (run.py profile): its lock (flock -w 21600; crescent ~/k3dgpu-bench.lock, else LOCK); the +# hold-start rule (load1 <= load1_max, and on crescent no aiperf process; 2-min polls, at most 60 +# min, then the hold runs flagged); the floors (crescent: / at most 97% used; desktop and wildct: +# 15 GB free on / and MemAvailable >= 20 GiB), or no hold (exit 5). It creates cluster +# , runs the arms (hold-run.sh) under `timeout -k 30 ` with a disk watchdog +# (and crescent's serving sampler), and deletes this campaign's clusters at the end, also on +# failure. Records: ${G}/results//hold/ (smoke: ${G}/smoke/-/). +# G campaign dir (default from the profile); H59_IMAGE the k3d-gpu node image, name:tag@sha256 +# LOCK the host lock (desktop; wildct's private inner lock); K3D_REAL the shim's real k3d +# SUDO_ASKPASS or K3D_GPU_RAPL_EXEC for energy.sh's RAPL reader +# H59_CPUINFO the cpuinfo file the host check reads (default /proc/cpuinfo) +set -uo pipefail +HOST=${1:?usage: hold.sh HOST HOLD [REPEATS]} HOLD=${2:?usage: hold.sh HOST HOLD [REPEATS]} REPEATS=${3:-} +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +RUN=${HERE}/run.py +python3 "${RUN}" profile "${HOST}" > /dev/null || exit 2 +pfield() { python3 "${RUN}" profile "${HOST}" | sed -n "s/^$1=//p"; } +ROW=$(pfield row) PREFIX=$(pfield prefix) CPU=$(pfield cpu) L1MAX=$(pfield load1_max) AIPERF=$(pfield aiperf) +DISK=$(pfield disk) MEMMIN=$(pfield mem_gib) CAP=$(pfield cap_s) SAMPLER=$(pfield sampler) +G=${G:-$(pfield g)} +G=${G/#\~/${HOME}} +order=$(python3 "${RUN}" hold "${HOLD}" ${REPEATS:+"${REPEATS}"}) || exit 2 +read -r MODE ARMS_RUN <<< "${order}" +read -r -a ARMS_RUN <<< "${ARMS_RUN}" +grep -m1 '^model name' "${H59_CPUINFO:-/proc/cpuinfo}" | grep -qF -- "${CPU}" \ + || { echo "hold.sh: this host's CPU is not the ${CPU} that ${ROW} registers" >&2; exit 2; } +[[ "${H59_IMAGE:-}" =~ ^[^@]+:[^@]+@sha256:[0-9a-f]{64}$ ]] \ + || { echo "hold.sh: set H59_IMAGE to the k3d-gpu node image as name:tag@sha256:" >&2; exit 2; } +cmp -s "${HERE}/k3d-shim.sh" "${G}/bin/k3d" || { echo "hold.sh: ${G}/bin/k3d is not bench/h59/k3d-shim.sh (install it first)" >&2; exit 2; } +python3 "${RUN}" models "${G}" >&2 || { echo "hold.sh: run bench/h59/prep.sh ${HOST} first" >&2; exit 2; } +RES=${G}/results/${ROW} +case ${HOLD} in smoke) R=${G}/smoke/${ROW}-$(date -u +%Y%m%dT%H%M%SZ) ;; *) R=${RES}/hold${HOLD} ;; esac +[ ! -e "${R}/rc" ] || { echo "hold.sh: hold ${HOLD} has run (${R}/rc); a failed hold counts, a spare repeats it" >&2; exit 2; } +if [ -n "${REPEATS}" ]; then + why=$(python3 "${RUN}" spare-ok "${RES}" "${REPEATS}") || { echo "hold.sh: spare ${HOLD} cannot repeat hold ${REPEATS}: ${why}" >&2; exit 2; } +fi +if [ "${HOLD}" != smoke ] && [ -f "${RES}/node-image" ] && [ "$(cat "${RES}/node-image")" != "${H59_IMAGE}" ]; then + echo "hold.sh: ${ROW}'s holds run node image $(cat "${RES}/node-image"), not ${H59_IMAGE}" >&2 + exit 2 +fi +case ${HOST} in crescent) LOCKF=${HOME}/k3dgpu-bench.lock ;; *) LOCKF=${LOCK:?set LOCK to this host\'s lock file} ;; esac +mkdir -p "${R}" "${G}/results" +export REPO G MODE H59_IMAGE H59_PROFILE=${HOST} H59_G=${G} H59_PREFIX=${PREFIX} HOLDNUM=${HOLD//[^0-9]/} +export PATH=${G}/bin:${PATH} CLUSTER=${PREFIX}${HOLD} KUBECONFIG=${R}/kubeconfig +[ "${HOLD}" != smoke ] || export SMOKE=1 +log() { echo "== ${ROW}-hold${HOLD} $* $(date -u +%FT%TZ)" | tee -a "${G}/results/campaign.log"; } +free_gb() { df -P -B1G / | awk 'NR==2 {print $4}'; } +used_pct() { df -P / | awk 'NR==2 {gsub("%", "", $5); print $5}'; } +memavail_gib() { awk '/MemAvailable/{print int($2/1048576)}' /proc/meminfo; } +aiperf_n() { docker top aiperf010 -eo pid,args 2>/dev/null | awk 'NR>1 && $0 !~ /sleep infinity/' | wc -l; } +low_disk() { if [ "${DISK}" = pct97 ]; then [ "$(used_pct)" -gt 97 ]; else [ "$(free_gb)" -lt 15 ]; fi; } +snapshot() { date -u +%FT%TZ; uptime; free -m; ps -eo pcpu,pid,user,comm --sort=-pcpu | head -11; df -P / "$(readlink -f "${G}")" | tail -2; } +# This campaign's clusters only (${PREFIX}*). +mine() { k3d cluster list -o json 2>/dev/null | python3 -c 'import json,sys; print(" ".join(c["name"] for c in json.load(sys.stdin) if c["name"].startswith(sys.argv[1])))' "${PREFIX}"; } + +log "waiting for the lock (mode ${MODE}, arms: ${ARMS_RUN[*]}${REPEATS:+, repeats hold ${REPEATS}})" +exec 9> "${LOCKF}" +flock -w 21600 9 || { log "lock not taken in 21600 s"; exit 4; } +log "lock taken" +waited=0 +while :; do + l1=$(cut -d' ' -f1 /proc/loadavg) a=0 + [ "${AIPERF}" = 0 ] || a=$(aiperf_n) + if awk -v l="${l1}" -v m="${L1MAX}" 'BEGIN {exit !(l > m)}' || [ "${a}" -gt 0 ]; then + if [ "${waited}" -ge 3600 ]; then log "FLAGGED: load1 ${l1}, aiperf ${a} after 60 min"; echo "load1 ${l1} aiperf ${a}" > "${R}/flagged"; break; fi + log "busy (load1 ${l1}, aiperf ${a}): waiting 120 s" + sleep 120 + waited=$(( waited + 120 )) + continue + fi + break +done +echo "${waited}" > "${R}/wait_s" +snapshot > "${R}/pre.txt" +if low_disk; then log "/ below the disk floor: no hold"; echo "disk: $(used_pct)% used, $(free_gb) GB free" > "${R}/no-hold"; exit 5; fi +if [ "${MEMMIN}" -gt 0 ] && [ "$(memavail_gib)" -lt "${MEMMIN}" ]; then + log "MemAvailable $(memavail_gib) GiB (< ${MEMMIN}): no hold"; echo "memavail $(memavail_gib) GiB" > "${R}/no-hold"; exit 5 +fi +[ "${HOLD}" = smoke ] || [ -f "${RES}/node-image" ] || echo "${H59_IMAGE}" > "${RES}/node-image" +# A spare names the hold it stands in for only once it runs (h59.py reads `repeats`). +[ -z "${REPEATS}" ] || echo "${REPEATS}" > "${R}/repeats" +left=$(mine) +[ -z "${left}" ] || { log "leftover clusters ${left}: deleting first"; k3d cluster delete ${left} > /dev/null 2>&1; } +bash "${HERE}/reproduce.sh" > "${R}/reproduce.txt" 2>&1 +sp="" +if [ "${SAMPLER}" = serving ]; then + python3 "${REPO}/bench/eatflow/crescent/serving.py" "${R}/serving.jsonl" & + sp=$! + export SERVING=${R}/serving.jsonl +fi +log "start: ${ARMS_RUN[*]}" +timeout -k 30 "${CAP}" bash "${HERE}/hold-run.sh" "${R}" "${ARMS_RUN[@]}" > "${R}/hold.log" 2>&1 & +cp=$! +# timeout(1) signals its whole process group; the watchdog stops that group below the disk floor. +( while kill -0 "${cp}" 2>/dev/null; do + sleep 30 + if low_disk; then echo "disk $(used_pct)% used, $(free_gb) GB free at $(date -u +%FT%TZ)" > "${R}/disk-abort"; kill -TERM "${cp}"; break; fi + done ) & +wd=$! +wait "${cp}" +rc=$? +kill "${wd}" ${sp} 2>/dev/null +[ -z "${sp}" ] || wait "${sp}" 2>/dev/null +left=$(mine) +[ -z "${left}" ] || k3d cluster delete ${left} > /dev/null 2>&1 +snapshot > "${R}/post.txt" +echo "${rc}" > "${R}/rc" +log "rc=${rc}; clusters deleted: ${left:-none}; now: $(mine)" +exit "${rc}" diff --git a/bench/h59/k3d-shim.sh b/bench/h59/k3d-shim.sh new file mode 100644 index 00000000..cb6fd100 --- /dev/null +++ b/bench/h59/k3d-shim.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# k3d-shim.sh: installed as ${G}/bin/k3d in front of the host's k3d (K3D_REAL, default +# /usr/bin/k3d). `k3d cluster create` of one of this campaign's clusters (H59_PREFIX) gets one more +# volume, the campaign's model directory at /var/apb-models in the server node, so models are read +# in place, never copied. It is writable: OVMS's --pull init container (arm O) writes graph.pbtxt +# beside the weights; arm L's pod mounts it read-only. Every other call passes through unchanged. +set -euo pipefail +REAL=${K3D_REAL:-/usr/bin/k3d} +G=${H59_G:-${HOME}/k3dgpu-h59} +if [ "${1:-}" = cluster ] && [ "${2:-}" = create ] && [ -n "${H59_PREFIX:-}" ] && [[ "${3:-}" == "${H59_PREFIX}"* ]]; then + exec "${REAL}" "$@" --volume="${G}/models:/var/apb-models@server:*" +fi +exec "${REAL}" "$@" diff --git a/bench/h59/prep.sh b/bench/h59/prep.sh new file mode 100644 index 00000000..845797a3 --- /dev/null +++ b/bench/h59/prep.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# prep.sh HOST: the H59-family model files on this host (EXPERIMENTS.md "H59 pre-registration", +# Model; "H59 amendment R1"). Preparations are not holds. +# 1. Downloads at the pinned commits, outside the lock, into ${G}/models: Google's QAT GGUF +# (gemma-4-26B_q4_0-it.gguf only) and OpenVINO's int4 IR (OVB); each file is checked against +# the Hub's LFS sha256 (bench/wildct/verify_models.py). The GGUF's directory is also reachable +# as gguf/@, the path L's render serves it from. OVB's directory and the OVMS +# compiled-model cache directory are mode 777: OVMS's --pull init container runs as uid 5000 +# and writes graph.pbtxt there at each start (H54 amendment R2's directory mode). +# 2. The QAT drafter, inside the host lock (flock -w 21600; crescent ~/k3dgpu-bench.lock, else +# LOCK) under `timeout -k 30 3600`: arm L rendered with no cluster (run.py prep-render), and +# its conversion Job's container (PR #182's path: full-b11960, its pip pins, convert_hf_to_gguf +# --outtype bf16, llama-quantize Q4_0) run once with docker against ${G}/models; the Job's +# spec is kept beside the GGUF as prep-job.json, which every hold's render must equal. +# 3. ${G}/weights.sha256: the sha256 of every file the arms read. +# G campaign dir (default from the profile); PY_HF a python with huggingface_hub +set -euo pipefail +HOST=${1:?usage: prep.sh HOST} +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +RUN=${HERE}/run.py +pfield() { python3 "${RUN}" profile "${HOST}" | sed -n "s/^$1=//p"; } +G=${G:-$(pfield g)} +G=${G/#\~/${HOME}} +M=${G}/models +PY_HF=${PY_HF:-python3} +export HF_HUB_DISABLE_TELEMETRY=1 REPO G +read -r GREPO GREV GFILE OVB OREV DDIR <<< "$(python3 -c 'import sys; sys.path.insert(0, sys.argv[1]); import run +print(run.GGUF_REPO, run.GGUF_REV, run.GGUF_FILE, run.OVB, run.OVB_REV, run.DRAFT_DIR)' "${HERE}")" +mkdir -p "${M}/gguf" "${G}/prep" +echo "downloads $(date -u +%FT%TZ); / free $(df -P -B1G / | awk 'NR==2{print $4}') GB" +nice -n 10 "${PY_HF}" - "${M}" "${GREPO}" "${GREV}" "${GFILE}" "${OVB}" "${OREV}" <<'EOF' +import sys +from huggingface_hub import hf_hub_download, snapshot_download +m, grepo, grev, gfile, ovb, orev = sys.argv[1:] +print(hf_hub_download(grepo, gfile, revision=grev, local_dir=f"{m}/{grepo}"), flush=True) +print(snapshot_download(ovb, revision=orev, local_dir=f"{m}/{ovb}"), flush=True) +EOF +echo "${GREV}" > "${M}/${GREPO}/.pinned-revision" +echo "${OREV}" > "${M}/${OVB}/.pinned-revision" +"${PY_HF}" "${REPO}/bench/wildct/verify_models.py" "${M}" --skip gemma-4-26B-it-mmproj.gguf "${GREPO}@${GREV}" "${OVB}@${OREV}" +ln -sfn "../${GREPO}" "${M}/gguf/${GREPO//\//__}@${GREV}" +mkdir -p "${M}/.ov_cache" +chmod 777 "${M}/${OVB}" "${M}/.ov_cache" + +case ${HOST} in crescent) LOCKF=${HOME}/k3dgpu-bench.lock ;; *) LOCKF=${LOCK:?set LOCK to this host\'s lock file} ;; esac +if [ ! -f "${M}/${DDIR}/manifest.json" ]; then + J=${G}/prep/drafter + rm -rf "${J}" + python3 "${RUN}" prep-render "${J}" + mapfile -t ARGV < <(python3 "${RUN}" docker-argv "${J}/job.json" "${M}" "${J}") + exec 9> "${LOCKF}" + echo "waiting for the lock $(date -u +%FT%TZ)" + flock -w 21600 9 + echo "lock taken $(date -u +%FT%TZ); converting the drafter ($(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["name"])' "${J}/job.json"))" + timeout -k 30 3600 "${ARGV[@]}" + cp "${J}/job.json" "${M}/${DDIR}/prep-job.json" + exec 9>&- +fi +python3 -c 'import json,sys; m=json.load(open(sys.argv[1])); print("drafter:", m["source"], m["quantize"], m["sha256"], m["bytes"])' "${M}/${DDIR}/manifest.json" +( cd "${M}" && find "${GREPO}/${GFILE}" "${OVB}" "${DDIR}/model.gguf" -type f ! -path '*/.cache/*' ! -name graph.pbtxt ! -name .pinned-revision -print0 \ + | sort -z | xargs -0 sha256sum ) > "${G}/weights.sha256" +python3 "${RUN}" models "${G}" +echo "done $(date -u +%FT%TZ); / free $(df -P -B1G / | awk 'NR==2{print $4}') GB" diff --git a/bench/h59/reproduce.sh b/bench/h59/reproduce.sh new file mode 100644 index 00000000..e7d6d838 --- /dev/null +++ b/bench/h59/reproduce.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# reproduce.sh: the Reproduce header of an H59-family hold (host, CPU, kernel, microcode, tools, +# repository commit, engine and node images, model revisions and file sha256). Each arm's +# rendered and staged manifests, server command line, CPUs and fit are in its own directory +# (agent.yaml, deploy.yaml, cpus.txt, fit.json). Read-only. +set -uo pipefail +G=${G:-${HOME}/k3dgpu-h59} +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +echo "# H59 Reproduce header (${H59_PROFILE:-?}), $(date -u +%FT%TZ)" +echo "host: $(hostname)" +echo "cpu: $(awk -F': ' '/^model name/{print $2; exit}' /proc/cpuinfo); $(nproc) CPUs; P-cores $(cat /sys/devices/cpu_core/cpus 2>/dev/null || echo none), E-cores $(cat /sys/devices/cpu_atom/cpus 2>/dev/null || echo none)" +echo "isa: amx $(grep -c amx_tile /proc/cpuinfo) avx512f $(grep -c avx512f /proc/cpuinfo) avx_vnni $(grep -c avx_vnni /proc/cpuinfo) (CPUs listing each)" +echo "microcode: $(awk -F': ' '/^microcode/{print $2; exit}' /proc/cpuinfo)" +echo "kernel: $(uname -r)" +echo "memory: $(awk '/MemTotal|MemAvailable|SwapTotal|SwapFree/{printf "%s %s kB; ", $1, $2}' /proc/meminfo)" +echo "governor: $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null); epp: $(cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference 2>/dev/null)" +echo "docker: $(docker version --format '{{.Server.Version}}' 2>/dev/null); k3d: $("${K3D_REAL:-/usr/bin/k3d}" version | head -1); kubectl: $(kubectl version --client 2>/dev/null | head -1)" +if [ -f "${REPO}/COMMIT" ]; then echo "repo: $(cat "${REPO}/COMMIT") (archive)"; else + echo "repo: $(git -C "${REPO}" rev-parse HEAD) $(git -C "${REPO}" status --porcelain | wc -l) uncommitted"; fi +echo "k3d-gpu: $("${REPO}/scripts/k3d-gpu" version 2>/dev/null | head -1)" +echo "images:" +python3 - "${HERE}" <<'EOF' +import sys +sys.path.insert(0, sys.argv[1]) +import run +print(f" ovms: {run.OVMS_IMAGE}\n llama.cpp: {run.LLAMA_CPU_IMAGE}\n llama.cpp (drafter conversion): {run.LLAMA_FULL_IMAGE}") +print(f"models: {run.GGUF_REPO}@{run.GGUF_REV} ({run.GGUF_FILE}), {run.OVB}@{run.OVB_REV}, " + f"{run.DRAFT_REPO}@{run.DRAFT_REV} (converted by prep.sh with the render's conversion Job)") +EOF +echo " node: ${H59_IMAGE:-unset}" +echo "drafter: $(python3 -c 'import json,sys; m=json.load(open(sys.argv[1])); print(m.get("source"), m.get("convert"), m.get("quantize"), m.get("sha256"), m.get("bytes"))' \ + "${G}/models/gguf/google__gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant@9537141506fe8875b3ed45b264af13580cb29166/manifest.json" 2>/dev/null)" +echo "file sha256:" +sed 's/^/ /' "${G}/weights.sha256" 2>/dev/null +echo "data: gsm8k-test-0-199.jsonl $(sha256sum "${REPO}/bench/h54/gsm8k-test-0-199.jsonl" | cut -c1-16); ttft-prompt.txt $(sha256sum "${REPO}/bench/h54/ttft-prompt.txt" | cut -c1-16)" diff --git a/bench/h59/run.py b/bench/h59/run.py new file mode 100644 index 00000000..b6adc1e3 --- /dev/null +++ b/bench/h59/run.py @@ -0,0 +1,563 @@ +#!/usr/bin/env python3 +"""run.py: the H59, H59d and H59w runner's tables and steps (EXPERIMENTS.md "H59 +pre-registration" and "H59 amendment R1"). The verdict is h59.py's; this file serves the shell +runner (hold.sh, hold-run.sh, arm.sh, prep.sh). + + run.py profile HOST the host profile as KEY=value lines (crescent, desktop, wildct) + run.py hold HOLD [REPEATS] "MODE ARM ARM" of a registered hold: 1-3 perf, 4 quality, + the spares 5 and 6 repeating hold REPEATS in its order, + smoke as hold 1 + run.py spare-ok RESULTS K exit 0 when hold K ran and one of its arms was not measured + run.py models G exit 0 when prep.sh left every file the arms read in G/models + run.py render ARM OUTDIR `k3d-gpu agent manifest --model` for the arm in the hold's + cluster (REPO, CLUSTER, KUBECONFIG, H59_PROFILE, G from the + environment), then stage and check: agent.yaml, agent.err, + render-env.txt, deploy.yaml, check.json, fit.json + run.py stage ARM IN OUT the staged render: models from the node's /var/apb-models + run.py check ARM HOST DEPLOY exit 1, naming each problem, when a staged render is not the + registered arm on that host + run.py job RENDER OUTDIR an L render's drafter conversion Job: convert.sh + job.json + (prep.sh runs it once per host; holds then skip the Job) + run.py gate RESULTS crescent's load verdict for `h59.py score --load GATE.json`: + H54's void rule over the N=4 phases of the performance holds + run.py prefix URL MODEL SEED OUT the prefix-reuse phase: 6 agents in sequence on one shared + prefix (H25's shape, bench/shared-prefix/probe.py) + +Python 3 standard library, except render/stage/check/job (PyYAML).""" + +import importlib.util +import json +import os +import statistics +import subprocess +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.abspath(os.path.join(HERE, "..", "..")) +sys.path.insert(0, HERE) +import h59 # noqa: E402 (hold_dirs, status: the scorer's own reading of the records) + +MODELS = "/var/apb-models" # the node path the k3d shim binds the campaign's G/models to +NS = "k3d-gpu-agent" +SERVED = "gemma" + +# The registered model files (H59 pre-registration, "Model"). +GGUF_REPO, GGUF_REV = "google/gemma-4-26B-A4B-it-qat-q4_0-gguf", "d1c082be9cf3c8a514acf63b8761f4b41935842e" +GGUF_FILE = "gemma-4-26B_q4_0-it.gguf" +DRAFT_REPO, DRAFT_REV = "google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant", "9537141506fe8875b3ed45b264af13580cb29166" +OVB, OVB_REV = "OpenVINO/gemma-4-26b-a4b-it-int4-ov", "4e5885c3f03844f5887982def09708bdc9f53f7a" + +# The engine images `agent manifest` pins at the runner's commit (PR #182's llama.cpp backend: +# server-b11960 on the CPU, full-b11960 for the drafter conversion; OVMS 2026.4.1). +OVMS_IMAGE = "openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58" +LLAMA_CPU_IMAGE = "ghcr.io/ggml-org/llama.cpp:server-b11960@sha256:8052c13edd987de8858f8150031e025ac3d66823f09cf3ec8220899c4abce220" +LLAMA_FULL_IMAGE = "ghcr.io/ggml-org/llama.cpp:full-b11960@sha256:7afc81bfe7a93de287324044fe86ccc70b1b7ac8832ed8e6ea658e11fb448ddf" + + +def slug(repo, rev): + """`k3d-gpu`'s model-cache directory name for repo@rev (gguf_slug).""" + return f"{repo.replace('/', '__')}@{rev}" + + +GGUF_PATH = f"/models/gguf/{slug(GGUF_REPO, GGUF_REV)}/{GGUF_FILE}" +DRAFT_DIR = f"gguf/{slug(DRAFT_REPO, DRAFT_REV)}" +DRAFT_PATH = f"/models/{DRAFT_DIR}/model.gguf" + +# Arms O and L as registered: `agent manifest --model` with K3D_GPU_AGENT_DEVICE=CPU, and for L +# K3D_GPU_AGENT_DRAFT=mtp. The rest of the render environment is fixed for both (COMMON_ENV). +ARMS = { + "O": {"model": "google/gemma-4-26B-A4B-it", "revision": OVB_REV, "env": {}, + "engine": "ovms", "deploy": "ovms", "svc": "ovms", "port": 8000, "api": "/v3"}, + "L": {"model": GGUF_REPO, "revision": GGUF_REV, "env": {"K3D_GPU_AGENT_DRAFT": "mtp"}, + "engine": "llama.cpp", "deploy": "llamacpp", "svc": "llamacpp", "port": 8000, "api": "/v1"}, +} +# K3D_GPU_AGENT_REVISION pins the fit's metadata read to the commit the weights were fetched at; +# the served name is the runner's; K3D_GPU_TUNE=off keeps a host's own tuned profile out. +COMMON_ENV = {"K3D_GPU_AGENT_DEVICE": "CPU", "K3D_GPU_AGENT_MODEL_NAME": SERVED, "K3D_GPU_TUNE": "off"} +# Variables a render never inherits from the shell that starts a hold: each one would change +# what `agent manifest` renders (backend, model, images, variant, ledger, fit mode, context). +SCRUB_PREFIXES = ("K3D_GPU_AGENT_", "K3D_GPU_LLAMA_") +SCRUB = ("K3D_GPU_MODEL_OPT", "K3D_GPU_MODEL_OPT_LEDGER", "K3D_GPU_AUTOQUANT", "K3D_GPU_FIT", "K3D_GPU_FIT_RUNG", + "K3D_GPU_FIT_FILE", "K3D_GPU_CONTEXT", "K3D_GPU_KUBECONFIG", "K3D_GPU_BALLOONS") + +# Holds: performance holds 1-3 in the rotated orders, the quality hold 4 in (L, O); the spares 5 +# and 6 repeat a failed hold's arms in its order (H59 pre-registration, "Holds and caps"). +ORDERS = {1: ["O", "L"], 2: ["L", "O"], 3: ["O", "L"], 4: ["L", "O"]} +MODES = {1: "perf", 2: "perf", 3: "perf", 4: "quality"} +SPARES = (5, 6) + +# The C1 kubelet thresholds (crescent amendment C1) and wildct's absolute form (rule W3). +C1 = "--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1%" +W3 = "--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<2Gi,imagefs.available<2Gi,nodefs.inodesFree<1%" + +# Host profiles. Each host keeps its own shared lock, start rule, floors, hold length and load +# rule (H59 pre-registration: H54's on crescent, H57d's on the desktop, the wildct +# pre-registration's on wildct). cpu: the model number /proc/cpuinfo must name (the row's CPU +# class); pcores: the P-core balloon's CPUs the render must request (0: no hybrid balloon). +PROFILES = { + "crescent": {"row": "H59", "cpu": "6741P", "g": "~/k3dgpu-h59", "prefix": "k3dgpu-g-h59-", + "lock": "~/k3dgpu-bench.lock", "load1_max": 12, "aiperf": 1, "disk": "pct97", "mem_gib": 0, + "cap_s": 2400, "arms_end_s": 2370, "vendors": "amx", "k3s_args": C1, + "sampler": "serving", "load_rule": "h54-void", "phases": "n1,n4,n16,ttft,prefix", + "pcores": 0}, + "desktop": {"row": "H59d", "cpu": "i9-13900T", "g": "~/k3dgpu-h59", "prefix": "k3dgpu-d-h59-", + "lock": "", "load1_max": 24, "aiperf": 0, "disk": "free15", "mem_gib": 20, + "cap_s": 4500, "arms_end_s": 4470, "vendors": "hybrid", "k3s_args": C1, + "sampler": "pcore", "load_rule": "h57d", "phases": "n1,n4,ttft,prefix", "pcores": 16}, + "wildct": {"row": "H59w", "cpu": "i7-13700", "g": "~/k3dgpu-bench-wildct/h59", "prefix": "k3dgpu-w-h59-", + "lock": "", "load1_max": 12, "aiperf": 0, "disk": "free15", "mem_gib": 20, + "cap_s": 4500, "arms_end_s": 4470, "vendors": "auto,-nvidia,-intel", "k3s_args": W3, + "sampler": "pcore", "load_rule": "h57d", "phases": "n1,n4,ttft,prefix", "pcores": 16}, +} +# Phases: (name, requests, concurrency, max_tokens) for bench/llm_load.py, H54's requests. +LOAD_PHASES = {"warmup": (3, 1, 256), "n1": (8, 1, 256), "n4": (16, 4, 256), "n16": (32, 16, 256), + "ttft": (4, 1, 16)} + + +def jload(path): + with open(path) as f: + return json.load(f) + + +def profile(host): + if host not in PROFILES: + raise SystemExit(f"run.py: unknown host profile {host!r} (crescent, desktop, wildct)") + return PROFILES[host] + + +def hold(n, repeats=None): + """(mode, arms) of a registered hold; SystemExit for anything else.""" + if n == "smoke" and repeats is None: + return "perf", ORDERS[1] + if n in [str(k) for k in ORDERS] and repeats is None: + return MODES[int(n)], ORDERS[int(n)] + if n in [str(k) for k in SPARES] and repeats in [str(k) for k in ORDERS]: + return MODES[int(repeats)], ORDERS[int(repeats)] + raise SystemExit(f"run.py hold: no registered hold {n!r}{' repeating ' + repr(repeats) if repeats else ''} " + "(1-3 perf, 4 quality, a spare 5 or 6 names the hold it repeats)") + + +def spare_ok(results, k): + """A spare repeats hold k only when hold k ran (its rc is recorded) and one of its arms was + not measured.""" + d = os.path.join(results, f"hold{k}") + if not os.path.exists(os.path.join(d, "rc")): + return False, f"hold {k} has not run" + mode, arms = hold(str(k)) + if all(h59.status(os.path.join(d, a)) == "measured" for a in arms): + return False, f"every arm of hold {k} was measured" + return True, f"hold {k} has an arm not measured" + + +def models_ready(g): + """The files the staged arms read, under G/models (prep.sh's layout); [] when all are there.""" + m = os.path.join(g, "models") + want = [os.path.join(m, GGUF_PATH[len("/models/"):]), os.path.join(m, DRAFT_DIR, "model.gguf"), + os.path.join(m, DRAFT_DIR, "manifest.json"), os.path.join(m, DRAFT_DIR, "prep-job.json"), + os.path.join(m, OVB, "openvino_model.xml"), os.path.join(m, OVB, "openvino_model.bin")] + return [p for p in want if not os.path.exists(p)] + + +# ---- YAML render steps --------------------------------------------------------------------------- + +def load_docs(path): + import yaml + with open(path) as f: + return [d for d in yaml.safe_load_all(f) if d] + + +def dump_docs(docs, path): + import yaml + with open(path, "w") as f: + yaml.safe_dump_all(docs, f, sort_keys=False) + + +def deployment(docs): + ds = [d for d in docs if d.get("kind") == "Deployment"] + if len(ds) != 1: + raise SystemExit(f"run.py: the render holds {len(ds)} Deployments, not 1") + return ds[0] + + +def _apb_stage(): + spec = importlib.util.spec_from_file_location("apb_stage", os.path.join(REPO, "bench", "apb", "stage.py")) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def stage(arm, docs): + """O: `stage.py ovms` (the PVC becomes the node's /var/apb-models; OVMS's --pull init + container writes graph.pbtxt there from this hold's own arguments, and finds the weights + prep.sh fetched at the pinned commit, so it downloads nothing). L: the PVC becomes the same + hostPath, mounted read-only; the drafter's conversion ConfigMap and Job and the init container + that waits for them are dropped (prep.sh ran that Job once, so each is a no-op), and so is + `--model-url` with its value (llama-server's download source; the GGUF is already at --model). + Image, every other argument, requests, limits and labels are as rendered.""" + docs = json.loads(json.dumps(docs)) + if arm == "O": + return _apb_stage().stage("ovms", docs) + if arm != "L": + raise SystemExit(f"run.py stage: unknown arm {arm}") + out = [] + for d in docs: + kind, name = d.get("kind"), (d.get("metadata") or {}).get("name", "") + if kind == "PersistentVolumeClaim" or (kind == "ConfigMap" and name == "gguf-convert") \ + or (kind == "Job" and name.startswith("gguf-convert-")): + continue + if kind == "Deployment": + ps = d["spec"]["template"]["spec"] + init = [c["name"] for c in ps.get("initContainers") or []] + if init not in ([], ["wait-gguf"]): + raise SystemExit(f"run.py stage L: unexpected init containers {init}") + ps.pop("initContainers", None) + for v in ps["volumes"]: + if v["name"] == "models": + v.pop("persistentVolumeClaim", None) + v["hostPath"] = {"path": MODELS, "type": "Directory"} + c = ps["containers"][0] + for m in c["volumeMounts"]: + if m["name"] == "models": + m["readOnly"] = True + a = c["args"] + if "--model-url" in a: + i = a.index("--model-url") + del a[i:i + 2] + out.append(d) + return out + + +def args_of(c): + return [str(x) for x in c.get("args") or []] + + +def value(args, opt): + """The argument after opt (llama-server takes an option's value from the next argument).""" + return args[args.index(opt) + 1] if opt in args and args.index(opt) + 1 < len(args) else None + + +def check(arm, host, docs): + """Problems that make a staged render not the registered arm on this host; [] when none.""" + p = profile(host) + bad = [] + try: + dep = deployment(docs) + except SystemExit as e: + return [str(e)] + want = ARMS[arm] + if dep["metadata"].get("name") != want["deploy"] or dep["metadata"].get("namespace") != NS: + bad.append(f"Deployment {dep['metadata'].get('namespace')}/{dep['metadata'].get('name')}, " + f"not {NS}/{want['deploy']}") + tpl = dep["spec"]["template"] + ps = tpl["spec"] + c = ps["containers"][0] + a = args_of(c) + ann = (tpl["metadata"].get("annotations") or {}) + if ann.get("k3d-gpu.io/agent-device") != "CPU": + bad.append(f"agent device {ann.get('k3d-gpu.io/agent-device')!r}, not CPU") + if arm == "O": + if c.get("image") != OVMS_IMAGE: + bad.append(f"image {c.get('image')}, not {OVMS_IMAGE}") + pull = [x for x in ps.get("initContainers") or [] if x["name"] == "pull"] + pa = args_of(pull[0]) if pull else [] + for need in ("--pull", f"--source_model={OVB}", "--target_device=CPU", "--task=text_generation", + f"--model_name={SERVED}"): + if need not in pa: + bad.append(f"pull container lacks {need}") + if any(x.startswith("--draft_") for x in pa + a): + bad.append("O renders a draft (registered: no draft)") + if f"--model_path=/models/{OVB}" not in a: + bad.append(f"server does not serve /models/{OVB}") + else: + if c.get("image") != LLAMA_CPU_IMAGE: + bad.append(f"image {c.get('image')}, not {LLAMA_CPU_IMAGE}") + for opt, v in (("--model", GGUF_PATH), ("--spec-draft-model", DRAFT_PATH), ("--spec-type", "draft-mtp"), + ("--alias", SERVED), ("--reasoning", "off")): + if value(a, opt) != v: + bad.append(f"{opt} {value(a, opt)!r}, not {v!r}") + for opt in ("--jinja",): + if opt not in a: + bad.append(f"no {opt}") + for opt in ("-c", "-np", "--threads", "--threads-batch"): + if value(a, opt) is None or not str(value(a, opt)).isdigit(): + bad.append(f"no numeric {opt}") + if "--model-url" in a: + bad.append("--model-url left in (a hold never downloads)") + if any(x.get("name") == "models" and not x.get("readOnly") for x in c.get("volumeMounts") or []): + bad.append("models mounted read-write") + if ps.get("initContainers"): + bad.append("init containers left in") + if p["pcores"] and value(a, "--threads") != str(p["pcores"] // 2): + bad.append(f"--threads {value(a, '--threads')}, not the {p['pcores'] // 2} physical P-cores") + if not any(v.get("name") == "models" and (v.get("hostPath") or {}).get("path") == MODELS for v in ps.get("volumes") or []): + bad.append(f"models volume is not hostPath {MODELS}") + if any(d.get("kind") == "PersistentVolumeClaim" for d in docs): + bad.append("a PersistentVolumeClaim is left in") + req = ((c.get("resources") or {}).get("requests") or {}) + lim = ((c.get("resources") or {}).get("limits") or {}) + if any("gpu" in k or "nvidia" in k for k in list(req) + list(lim)): + bad.append("a GPU resource is requested") + label = (tpl["metadata"].get("labels") or {}).get("k3d-gpu.io/cores") + if p["pcores"]: + # The P-core balloon (H57dp's deviation: a render that misses the hold's cluster gets no + # hybrid node, so no label and 1 CPU). + if label != "performance": + bad.append(f"pod label k3d-gpu.io/cores {label!r}, not performance (the P-core balloon)") + if str(req.get("cpu")) != str(p["pcores"]): + bad.append(f"CPU request {req.get('cpu')!r}, not the {p['pcores']} P-core CPUs") + if arm == "L" and value(a, "--threads-batch") != str(p["pcores"]): + bad.append(f"--threads-batch {value(a, '--threads-batch')}, not {p['pcores']}") + elif label is not None: + bad.append(f"pod label k3d-gpu.io/cores {label!r} on a host with no hybrid balloon") + return bad + + +def fit_record(arm, docs): + """The fit the arm ran with, from the staged render.""" + dep = deployment(docs) + tpl = dep["spec"]["template"] + ps = tpl["spec"] + c = ps["containers"][0] + a = args_of(c) + rec = {"arm": arm, "fit": (tpl["metadata"].get("annotations") or {}).get("k3d-gpu.io/agent-fit"), + "requests": (c.get("resources") or {}).get("requests"), "limits": (c.get("resources") or {}).get("limits"), + "image": c.get("image"), "args": a} + if arm == "O": + pull = [x for x in ps.get("initContainers") or [] if x["name"] == "pull"] + pa = args_of(pull[0]) if pull else [] + rec["pull_args"] = pa + for k in ("max_num_seqs", "cache_size", "enable_prefix_caching"): + v = [x.split("=", 1)[1] for x in pa if x.startswith(f"--{k}=")] + rec[k] = v[0] if v else None + else: + for k in ("-c", "-np", "-ngl", "--threads", "--threads-batch", "--load-mode"): + rec[k.lstrip("-")] = value(a, k) + return rec + + +def job_spec(docs): + """The drafter conversion Job of an L render (image, command, args, env) and its script.""" + jobs = [d for d in docs if d.get("kind") == "Job" and d["metadata"]["name"].startswith("gguf-convert-")] + cms = [d for d in docs if d.get("kind") == "ConfigMap" and d["metadata"]["name"] == "gguf-convert"] + if len(jobs) != 1 or len(cms) != 1: + raise SystemExit(f"run.py job: the render holds {len(jobs)} conversion Jobs and {len(cms)} ConfigMaps, not 1 and 1") + c = jobs[0]["spec"]["template"]["spec"]["containers"][0] + spec = {"name": jobs[0]["metadata"]["name"], "image": c["image"], "command": c["command"], + "args": [str(x) for x in c["args"]], "env": {e["name"]: str(e["value"]) for e in c.get("env") or []}, + "resources": c.get("resources")} + return spec, cms[0]["data"]["convert.sh"] + + +def write_job(render_yaml, outdir): + spec, script = job_spec(load_docs(render_yaml)) + os.makedirs(outdir, exist_ok=True) + with open(os.path.join(outdir, "job.json"), "w") as f: + json.dump(spec, f, indent=1) + with open(os.path.join(outdir, "convert.sh"), "w") as f: + f.write(script) + return spec + + +def prep_render(outdir): + """prep.sh's L render, with no cluster (the conversion Job does not depend on one): a scratch + kubeconfig keeps every kubectl call off the host's clusters. Writes agent.yaml, agent.err, + job.json and convert.sh.""" + repo = os.environ.get("REPO", REPO) + os.makedirs(outdir, exist_ok=True) + env = render_env("L", os.environ, "h59-prep") + env["KUBECONFIG"] = os.path.join(outdir, "kubeconfig") + open(env["KUBECONFIG"], "a").close() + cmd = [os.path.join(repo, "scripts", "k3d-gpu"), "agent", "manifest", "--model", ARMS["L"]["model"]] + with open(os.path.join(outdir, "agent.yaml"), "w") as fo, open(os.path.join(outdir, "agent.err"), "w") as fe: + rc = subprocess.run(cmd, env=env, stdout=fo, stderr=fe).returncode + if rc != 0: + raise SystemExit(f"prep render failed: agent manifest exited {rc}") + return write_job(os.path.join(outdir, "agent.yaml"), outdir) + + +def docker_argv(spec, models, scripts): + """`docker run` of the conversion Job's container on this host: its image, command, arguments + and environment, the campaign's models at /models and the Job's script at /scripts, run as + the campaign user, under the Job's memory limit.""" + if spec["image"] != LLAMA_FULL_IMAGE: + raise SystemExit(f"run.py: the conversion Job runs {spec['image']}, not {LLAMA_FULL_IMAGE}") + if spec["command"] != ["bash", "/scripts/convert.sh"]: + raise SystemExit(f"run.py: unexpected conversion command {spec['command']}") + argv = ["docker", "run", "--rm", "-u", f"{os.getuid()}:{os.getgid()}"] + lim = ((spec.get("resources") or {}).get("limits") or {}).get("memory") + if lim: + argv += ["--memory", lim.replace("Gi", "g")] + for k, v in spec["env"].items(): + argv += ["-e", f"{k}={v}"] + argv += ["-v", f"{models}:/models", "-v", f"{scripts}:/scripts:ro", "--entrypoint", "bash", spec["image"], + "/scripts/convert.sh", *spec["args"]] + return argv + + +def render_env(arm, environ, cluster): + env = {k: v for k, v in environ.items() + if not k.startswith(SCRUB_PREFIXES) and k not in SCRUB} + env.update(COMMON_ENV) + env.update(ARMS[arm]["env"]) + env["K3D_GPU_AGENT_REVISION"] = ARMS[arm]["revision"] + # The hold's cluster: `agent manifest` reads the node's hybrid label and allocatable CPU + # through context k3d-${K3D_GPU_CLUSTER}; without it that is k3d-gpu, which a hold's own + # kubeconfig does not hold (H57dp's Od deviation). + env["K3D_GPU_CLUSTER"] = cluster + return env + + +def render(arm, out): + """Render, stage, check and record one arm in the hold's cluster; exit 1 with the reason + when the render fails or is not the registered arm.""" + host, cluster = os.environ.get("H59_PROFILE", ""), os.environ.get("CLUSTER", "") + repo = os.environ.get("REPO", REPO) + if not cluster: + raise SystemExit("run.py render: CLUSTER (the hold's cluster) is not set") + profile(host) + os.makedirs(out, exist_ok=True) + env = render_env(arm, os.environ, cluster) + with open(os.path.join(out, "render-env.txt"), "w") as f: + f.writelines(f"{k}={env[k]}\n" for k in sorted(env) if k.startswith("K3D_GPU_") or k == "KUBECONFIG") + cmd = [os.path.join(repo, "scripts", "k3d-gpu"), "agent", "manifest", "--model", ARMS[arm]["model"]] + with open(os.path.join(out, "agent.yaml"), "w") as fo, open(os.path.join(out, "agent.err"), "w") as fe: + rc = subprocess.run(cmd, env=env, stdout=fo, stderr=fe).returncode + if rc != 0: + raise SystemExit(f"render failed: agent manifest exited {rc}") + docs = load_docs(os.path.join(out, "agent.yaml")) + rec = {"arm": arm, "host": host, "problems": []} + if arm == "L": + spec, _ = job_spec(docs) + rec["job"] = spec + g = os.path.expanduser(os.environ.get("G", "")) + pj = os.path.join(g, "models", DRAFT_DIR, "prep-job.json") + if not g or not os.path.exists(pj): + rec["problems"].append(f"no prep-job.json for the drafter at {pj}") + elif jload(pj) != spec: + rec["problems"].append("the render's drafter conversion Job differs from the one prep.sh ran") + staged = stage(arm, docs) + dump_docs(staged, os.path.join(out, "deploy.yaml")) + rec["problems"] += check(arm, host, staged) + fit = fit_record(arm, staged) + seqs = fit.get("max_num_seqs") if arm == "O" else fit.get("np") + # Reported, not gated: the arm runs at the fit's sequences, as registered. + rec["notes"] = [f"the fit gives {seqs} sequences (< 4): the N=4 phase queues"] \ + if seqs is not None and str(seqs).isdigit() and int(seqs) < 4 else [] + with open(os.path.join(out, "fit.json"), "w") as f: + json.dump(fit, f, indent=1) + with open(os.path.join(out, "check.json"), "w") as f: + json.dump(rec, f, indent=1) + if rec["problems"]: + raise SystemExit("render not the registered arm: " + "; ".join(rec["problems"])) + + +# ---- crescent's load rule ------------------------------------------------------------------------ + +def gate(results): + """H54's rule on crescent (amendment R4): a phase whose idle window read non-campaign CPU + above 4 CPU-s/s, or that saw an aiperf process, is void and re-run once. The row's load is + unmatched when an N=4 phase of a performance hold (a spare standing in for the hold it + repeats) was void on both tries, or carries no void check.""" + hd = h59.hold_dirs(results) + void, checked = [], 0 + for k in h59.PERF_HOLDS: + for arm in ("O", "L"): + ad = os.path.join(hd[k], arm) + if h59.status(ad) != "measured": + continue + p = os.path.join(ad, "phases.jsonl") + tries = [] + if os.path.exists(p): + with open(p) as f: + tries = [r for r in (json.loads(x) for x in f if x.strip()) if r.get("phase") == "n4"] + last = tries[-1] if tries else None + if not last or "check" not in last: + void.append(f"hold {k} {arm}: no void check for N=4") + elif last["check"].get("void"): + void.append(f"hold {k} {arm}: N=4 void on {len(tries)} tries ({last['check'].get('reason')})") + checked += 1 + if void: + return {"matched": False, "reason": "; ".join(void), "rule": "H54's void rule over the N=4 phases", + "checked": checked} + return {"matched": True, "reason": f"no N=4 phase void after its re-run ({checked} checked)", + "rule": "H54's void rule over the N=4 phases", "checked": checked} + + +# ---- the prefix-reuse phase ---------------------------------------------------------------------- + +def _probe(): + spec = importlib.util.spec_from_file_location("h25_probe", os.path.join(REPO, "bench", "shared-prefix", "probe.py")) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def prefix(url, model, seed, out): + """H25's phase S without its control: agents a1-a6 in sequence, each the byte-identical + system text and 12 tool schemas plus its own 96-token message, max_tokens 8, thinking off, + streamed. The score is the median first-chunk TTFT of agents 2-6.""" + probe = _probe() + probe.MODEL = model + p = probe.gen(seed) + chat = url.rstrip("/") + "/chat/completions" + agents = [dict(probe.request(chat, p["system"], u, p["tools"], 8), agent=f"a{n + 1}") for n, u in enumerate(p["a"])] + added = [a["ttft"] for a in agents[1:] if a["ttft"] is not None] + rec = {"seed": seed, "model": model, "agents": agents, "first_ttft_s": agents[0]["ttft"], + "added_ttft_median_s": statistics.median(added) if len(added) == 5 else None, + "prompt_tokens": sorted({str(a["prompt_tokens"]) for a in agents})} + with open(out, "w") as f: + json.dump(rec, f, indent=1) + return rec + + +def main(argv): + if not argv: + raise SystemExit(__doc__) + cmd, a = argv[0], argv[1:] + if cmd == "profile": + for k, v in profile(a[0]).items(): + print(f"{k}={v}") + elif cmd == "arms": + print(json.dumps({"arms": ARMS, "orders": ORDERS, "modes": MODES, "common_env": COMMON_ENV, + "phases": LOAD_PHASES}, indent=1)) + elif cmd == "field": + print(ARMS[a[0]][a[1]]) + elif cmd == "hold": + mode, arms = hold(a[0], a[1] if len(a) > 1 else None) + print(mode, *arms) + elif cmd == "spare-ok": + ok, why = spare_ok(a[0], a[1]) + print(why) + return 0 if ok else 1 + elif cmd == "models": + missing = models_ready(os.path.expanduser(a[0])) + for m in missing: + print(f"missing {m}") + return 1 if missing else 0 + elif cmd == "render": + render(a[0], a[1]) + elif cmd == "stage": + dump_docs(stage(a[0], load_docs(a[1])), a[2]) + elif cmd == "check": + bad = check(a[0], a[1], load_docs(a[2])) + for b in bad: + print(b) + return 1 if bad else 0 + elif cmd == "job": + write_job(a[0], a[1]) + elif cmd == "prep-render": + prep_render(a[0]) + elif cmd == "docker-argv": + print("\n".join(docker_argv(jload(a[0]), a[1], a[2]))) + elif cmd == "gate": + print(json.dumps(gate(a[0]), indent=1)) + elif cmd == "prefix": + rec = prefix(a[0], a[1], int(a[2]), a[3]) + print(json.dumps({"added_ttft_median_s": rec["added_ttft_median_s"], "first_ttft_s": rec["first_ttft_s"]})) + else: + raise SystemExit(f"run.py: unknown command {cmd}") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bench/h59/test_run.py b/bench/h59/test_run.py new file mode 100644 index 00000000..cc225cd8 --- /dev/null +++ b/bench/h59/test_run.py @@ -0,0 +1,729 @@ +#!/usr/bin/env python3 +"""Tests for the H59-family runner (bench/h59: run.py, arm.sh, hold.sh, hold-run.sh, prep.sh; +EXPERIMENTS.md "H59 amendment R1"). Offline: `k3d-gpu agent manifest` renders against the fit +fixtures (tests/fixtures/fit, tests/fixtures/hf) with a kubectl stub that answers only for the +hold's own context, so a render that misses the hold's cluster is caught; arm.sh runs against a +local OpenAI-compatible server with kubectl and docker stubbed, and the records it writes are +scored by h59.py, so the runner's N=4 holds feed the registered bootstrap-ratio scorer end to +end. No cluster, no network.""" +import http.server +import json +import os +import shutil +import subprocess +import sys +import tempfile +import threading +import time +import unittest + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.abspath(os.path.join(HERE, "..", "..")) +sys.path.insert(0, HERE) +import h59 # noqa: E402 +import run # noqa: E402 + +FIX = os.path.join(ROOT, "tests", "fixtures", "fit") +HOSTS = {"desktop": "desktop-i9-nogpu.json", "wildct": "desktop-i9-nogpu.json", "crescent": "xeon6741p-4xb70.json"} +NODE_IMAGE = "cryptoandcoffee/k3d-gpu:0.1.80@sha256:" + "a" * 64 + +KUBECTL = r'''#!/usr/bin/env python3 +"""kubectl stand-in: the launcher's calls name --context and are answered only for +k3d-${STUB_CLUSTER} ("context was not found" otherwise, as a hold's own kubeconfig answers); +the runner's own calls name no context.""" +import os, shutil, sys +a = sys.argv[1:] +ctx = None +while a and a[0] == "--context": + ctx, a = a[1], a[2:] +d = os.environ["STUB_DIR"] +with open(os.path.join(d, "kubectl.log"), "a") as f: + f.write(f"ctx={ctx} {' '.join(a)}\n") +if ctx is not None and ctx != "k3d-" + os.environ.get("STUB_CLUSTER", ""): + sys.exit(f"error: context was not found for specified context: {ctx}") +s = " ".join(a) +if s == "get nodes -l k3d-gpu.io/hybrid-cpu=true -o name": + if os.environ.get("STUB_HYBRID"): + print("node/k3d-x-server-0") +elif "allocatable" in s: + print(os.environ.get("STUB_ALLOC", "cpu 32")) +elif a[:2] == ["apply", "-f"]: + shutil.copy(a[2], os.path.join(d, "applied.yaml")) + sys.exit(int(os.environ.get("STUB_APPLY_RC", "0"))) +elif "port-forward" in a: + os.execvp("sleep", ["sleep", "600"]) +elif "{.items[*].status.phase}" in s: + print("Running") +elif "{.items[0].metadata.name}" in s: + print("pod-0", end="") +elif "{.spec.containers[0].name}" in s: + print("server", end="") +elif "{.spec.containers[0].image}" in s: + print("image@sha256:0 imageid", end="") +elif "-o json" in s: + print("{}") +elif "exec" in a: + print("cpuset 0-15\nCpus_allowed_list:\t0-15") +''' + +DOCKER = r'''#!/bin/sh +case "$1" in + inspect) echo deadbeef ;; + top) exit 1 ;; +esac +exit 0 +''' + + +def jl(path): + with open(path) as f: + return json.load(f) + + +def rd(path): + with open(path) as f: + return f.read() + + +def put(path, text, mode=None): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w") as f: + f.write(text) + if mode: + os.chmod(path, mode) + + +class Fake(http.server.ThreadingHTTPServer): + """OpenAI-compatible stand-in: /v1 (L) and /v3 (O) stream at their own inter-token gap; a + non-streamed chat answers one fixed sentence. Every request body is kept.""" + daemon_threads = True + + def __init__(self, itl): + self.itl, self.bodies = itl, [] + super().__init__(("127.0.0.1", 0), Handler) + + +class Handler(http.server.BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def log_message(self, *a): + pass + + def reply(self, obj): + raw = json.dumps(obj).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + self.wfile.write(raw) + + def do_GET(self): + if self.path.endswith("/models"): + return self.reply({"object": "list", "data": [{"id": "gemma"}]}) + self.send_error(404) + + def do_POST(self): + body = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0)))) + self.server.bodies.append((self.path, body)) + itl = self.server.itl[self.path.split("/")[1]] + n = body.get("max_tokens") or 16 + if not body.get("stream"): + return self.reply({"choices": [{"index": 0, "message": {"role": "assistant", "content": "The answer is 0."}, + "finish_reason": "stop"}], "usage": {"prompt_tokens": 9, "completion_tokens": 5}}) + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Transfer-Encoding", "chunked") + self.end_headers() + + def chunk(obj): + raw = f"data: {obj if isinstance(obj, str) else json.dumps(obj)}\n\n".encode() + self.wfile.write(f"{len(raw):x}\r\n".encode() + raw + b"\r\n") + self.wfile.flush() + chunk({"choices": [{"index": 0, "delta": {"role": "assistant", "content": None}, "finish_reason": None}]}) + for i in range(n): + time.sleep(itl) + chunk({"choices": [{"index": 0, "delta": {"content": f"t{i} "}, "finish_reason": "length" if i == n - 1 else None}]}) + if (body.get("stream_options") or {}).get("include_usage"): + chunk({"choices": [], "usage": {"prompt_tokens": 1600, "completion_tokens": n}}) + chunk("[DONE]") + self.wfile.write(b"0\r\n\r\n") + + +def serve(itl): + srv = Fake(itl) + threading.Thread(target=srv.serve_forever, daemon=True).start() + return srv + + +class Box: + """A scratch host: fit fixtures, a sysroot (hybrid i9 or the Xeon), kubectl and docker stubs, + a fake RAPL zone, the campaign dir G with prep's drafter Job record.""" + + def __init__(self, host, cluster="k3dgpu-d-h59-1", stub_cluster=None): + self.d = tempfile.mkdtemp(prefix="h59-") + d = self.d + self.host, self.cluster = host, cluster + put(os.path.join(d, "bin", "kubectl"), KUBECTL, 0o755) + put(os.path.join(d, "bin", "docker"), DOCKER, 0o755) + os.makedirs(os.path.join(d, "cache")) + for n in os.listdir(os.path.join(FIX, "models")): + src = os.path.join(FIX, "models", n) + os.symlink(src, os.path.join(d, "cache", f"{n}@main")) + rev = jl(os.path.join(src, "weights.json")).get("source", "").partition("@")[2] + if rev: + os.symlink(src, os.path.join(d, "cache", f"{n}@{rev}")) + # O's render pins K3D_GPU_AGENT_REVISION to OVB's commit, and the launcher also reads the + # requested base repo's format at that revision (agent_model_format). On a host that read + # finds no such commit and the format is unknown; here it reads the base's own metadata, + # which is not GGUF either, so the render takes the same path without the network. + os.symlink(os.path.join(FIX, "models", "google__gemma-4-26B-A4B-it"), + os.path.join(d, "cache", f"google__gemma-4-26B-A4B-it@{run.OVB_REV}")) + sysd = os.path.join(d, "root", "sys", "devices") + hybrid = run.PROFILES[host]["pcores"] > 0 + for c in range(32 if hybrid else 96): + if hybrid: + sib = f"{c // 2 * 2}-{c // 2 * 2 + 1}" if c < 16 else str(c) + else: + sib = f"{c % 48},{c % 48 + 48}" + put(os.path.join(sysd, "system", "cpu", f"cpu{c}", "topology", "core_cpus_list"), sib + "\n") + if hybrid: + put(os.path.join(sysd, "cpu_core", "cpus"), "0-15\n") + put(os.path.join(sysd, "cpu_atom", "cpus"), "16-31\n") + rapl = os.path.join(d, "powercap", "intel-rapl:0") + put(os.path.join(rapl, "name"), "package-0\n") + put(os.path.join(rapl, "energy_uj"), "123456789\n") + put(os.path.join(rapl, "max_energy_range_uj"), "262143328850\n") + os.makedirs(os.path.join(d, "home")) + os.makedirs(os.path.join(d, "stub")) + put(os.path.join(d, "kubeconfig"), "") + self.g = os.path.join(d, "g") + self.env = { + "PATH": os.path.join(d, "bin") + ":/usr/local/bin:/usr/bin:/bin", "HOME": os.path.join(d, "home"), + "LANG": "C.UTF-8", "K3D_GPU_SYSROOT": os.path.join(d, "root"), "K3D_GPU_FIT_CACHE": os.path.join(d, "cache"), + "XDG_STATE_HOME": os.path.join(d, "state"), "XDG_CACHE_HOME": os.path.join(d, "xdg"), + "K3D_GPU_HF_FIXTURES": os.path.join(ROOT, "tests", "fixtures", "hf"), + "K3D_GPU_AUTOQUANT_CACHE": os.path.join(d, "no-autoquant.json"), + "K3D_GPU_FIT_HOST": os.path.join(FIX, "hosts", HOSTS[host]), + "KUBECONFIG": os.path.join(d, "kubeconfig"), "STUB_DIR": os.path.join(d, "stub"), + "STUB_CLUSTER": stub_cluster or cluster, "STUB_ALLOC": "cpu 32" if hybrid else "cpu 64", + "REPO": ROOT, "G": self.g, "CLUSTER": cluster, "H59_PROFILE": host, + "ENERGY_POWERCAP": os.path.join(d, "powercap"), "H59_IDLE_S": "1", + } + if hybrid: + self.env["STUB_HYBRID"] = "1" + + def prep_job(self): + """prep.sh's drafter record: the conversion Job of an L render with no cluster.""" + out = os.path.join(self.d, "prep") + r = subprocess.run([sys.executable, os.path.join(HERE, "run.py"), "prep-render", out], env=self.env, + capture_output=True, text=True) + assert r.returncode == 0, r.stderr + rd(os.path.join(out, "agent.err")) + dst = os.path.join(self.g, "models", run.DRAFT_DIR) + os.makedirs(dst, exist_ok=True) + shutil.copy(os.path.join(out, "job.json"), os.path.join(dst, "prep-job.json")) + return jl(os.path.join(out, "job.json")) + + def render(self, arm, out, **extra): + env = dict(self.env, **extra) + return subprocess.run([sys.executable, os.path.join(HERE, "run.py"), "render", arm, out], env=env, + capture_output=True, text=True) + + def close(self): + shutil.rmtree(self.d, ignore_errors=True) + + +class Tables(unittest.TestCase): + def test_orders_as_registered(self): + self.assertEqual(run.ORDERS, {1: ["O", "L"], 2: ["L", "O"], 3: ["O", "L"], 4: ["L", "O"]}) + self.assertEqual([run.hold(str(k)) for k in (1, 2, 3, 4)], + [("perf", ["O", "L"]), ("perf", ["L", "O"]), ("perf", ["O", "L"]), ("quality", ["L", "O"])]) + self.assertEqual(run.hold("5", "2"), ("perf", ["L", "O"])) + self.assertEqual(run.hold("6", "4"), ("quality", ["L", "O"])) + self.assertEqual(run.hold("smoke"), ("perf", ["O", "L"])) + for bad in (("5", None), ("7", None), ("01", None), ("1", "2"), ("5", "5"), ("0", None), ("smoke", "1")): + with self.assertRaises(SystemExit): + run.hold(*bad) + + def test_profiles_name_each_rows_cpu_class(self): + rows = {} + for line in rd(os.path.join(ROOT, "EXPERIMENTS.md")).splitlines(): + if line.startswith("| H59"): + rows[line.split("|")[1].strip()] = line + for host, p in run.PROFILES.items(): + self.assertIn(p["cpu"], rows[p["row"]], host) + self.assertEqual({p["row"] for p in run.PROFILES.values()}, {"H59", "H59d", "H59w"}) + + def test_host_rules(self): + c, d, w = run.PROFILES["crescent"], run.PROFILES["desktop"], run.PROFILES["wildct"] + # N=16 on crescent only; H54's void rule there, H57d's load gate on the two clients. + self.assertIn("n16", c["phases"].split(",")) + self.assertNotIn("n16", d["phases"].split(",")) + self.assertNotIn("n16", w["phases"].split(",")) + self.assertEqual((c["load_rule"], d["load_rule"], w["load_rule"]), ("h54-void", "h57d", "h57d")) + # Hold-start load1: H54's 12, H57d's 24, wildct's 12; caps: H54's 2400 s, H57d's 4500 s. + self.assertEqual((c["load1_max"], d["load1_max"], w["load1_max"]), (12, 24, 12)) + self.assertEqual((c["cap_s"], d["cap_s"], w["cap_s"]), (2400, 4500, 4500)) + self.assertEqual((c["arms_end_s"], d["arms_end_s"]), (2370, 4470)) + self.assertEqual((c["vendors"], d["vendors"], w["vendors"]), ("amx", "hybrid", "auto,-nvidia,-intel")) + self.assertEqual(w["k3s_args"], run.W3) + self.assertTrue(c["prefix"].startswith("k3dgpu-g-") and d["prefix"].startswith("k3dgpu-d-") + and w["prefix"].startswith("k3dgpu-w-")) + self.assertEqual((c["pcores"], d["pcores"], w["pcores"]), (0, 16, 16)) + + def test_render_env_scrubs_and_pins(self): + shell = {"K3D_GPU_AGENT_BACKEND": "vllm", "K3D_GPU_AGENT_MODEL": "x/y", "K3D_GPU_LLAMA_CPU_IMAGE": "i", + "K3D_GPU_MODEL_OPT": "off", "K3D_GPU_FIT": "off", "K3D_GPU_CONTEXT": "k3d-gpu", "PATH": "/bin", + "K3D_GPU_FIT_HOST": "/h.json", "KUBECONFIG": "/k"} + o = run.render_env("O", shell, "k3dgpu-d-h59-2") + lo = run.render_env("L", shell, "k3dgpu-d-h59-2") + for e in (o, lo): + for k in ("K3D_GPU_AGENT_BACKEND", "K3D_GPU_AGENT_MODEL", "K3D_GPU_LLAMA_CPU_IMAGE", "K3D_GPU_MODEL_OPT", + "K3D_GPU_FIT", "K3D_GPU_CONTEXT"): + self.assertNotIn(k, e) + self.assertEqual(e["K3D_GPU_CLUSTER"], "k3dgpu-d-h59-2") + self.assertEqual((e["K3D_GPU_AGENT_DEVICE"], e["K3D_GPU_TUNE"], e["PATH"], e["KUBECONFIG"]), ("CPU", "off", "/bin", "/k")) + self.assertEqual(o["K3D_GPU_AGENT_REVISION"], run.OVB_REV) + self.assertNotIn("K3D_GPU_AGENT_DRAFT", o) + self.assertEqual((lo["K3D_GPU_AGENT_REVISION"], lo["K3D_GPU_AGENT_DRAFT"]), (run.GGUF_REV, "mtp")) + + def test_docker_argv_runs_the_jobs_container(self): + spec = {"name": "gguf-convert-x", "image": run.LLAMA_FULL_IMAGE, "command": ["bash", "/scripts/convert.sh"], + "args": [run.DRAFT_REPO, run.DRAFT_REV, "/models/" + run.DRAFT_DIR, "4294967296", "Q4_0", "--outtype", "bf16"], + "env": {"GGUF_CONVERT_PIP": "transformers==5.21.0 x==1", "HOME": "/tmp"}, + "resources": {"limits": {"memory": "64Gi"}}} + a = run.docker_argv(spec, "/g/models", "/g/prep/drafter") + self.assertEqual(a[a.index("--memory") + 1], "64g") + self.assertIn("GGUF_CONVERT_PIP=transformers==5.21.0 x==1", a) + self.assertIn("/g/models:/models", a) + self.assertEqual(a[-8:], ["/scripts/convert.sh"] + spec["args"]) + with self.assertRaises(SystemExit): + run.docker_argv(dict(spec, image="ghcr.io/ggml-org/llama.cpp:full-b11749"), "/m", "/s") + + +class Render(unittest.TestCase): + """`agent manifest --model` through run.py render, offline, for each host profile.""" + + def setUp(self): + self.box = Box("desktop") + self.job = self.box.prep_job() + self.out = os.path.join(self.box.d, "out") + + def tearDown(self): + self.box.close() + + def docs(self, arm, name="deploy.yaml"): + return run.load_docs(os.path.join(self.out, arm, name)) + + def test_both_arms_render_on_the_pcore_balloon(self): + for arm in ("O", "L"): + r = self.box.render(arm, os.path.join(self.out, arm)) + self.assertEqual(r.returncode, 0, r.stderr) + tpl = run.deployment(self.docs(arm))["spec"]["template"] + c = tpl["spec"]["containers"][0] + self.assertEqual(tpl["metadata"]["labels"]["k3d-gpu.io/cores"], "performance") + self.assertEqual(str(c["resources"]["requests"]["cpu"]), "16") + self.assertEqual(jl(os.path.join(self.out, arm, "check.json"))["problems"], []) + env = rd(os.path.join(self.out, arm, "render-env.txt")) + self.assertIn("K3D_GPU_CLUSTER=k3dgpu-d-h59-1\n", env) + log = rd(os.path.join(self.box.d, "stub", "kubectl.log")) + self.assertIn("ctx=k3d-k3dgpu-d-h59-1 get nodes -l k3d-gpu.io/hybrid-cpu=true -o name", log) + + def test_a_render_that_misses_the_holds_cluster_is_refused(self): + # H57dp's Od deviation: the launcher's context is not the hold's, the node queries fail, + # and the render has no P-core label and 1 CPU. The check stops the arm before apply. + box = Box("desktop", stub_cluster="some-other-cluster") + try: + box.prep_job() + r = box.render("O", os.path.join(box.d, "o")) + self.assertNotEqual(r.returncode, 0) + self.assertIn("not performance (the P-core balloon)", r.stderr) + self.assertIn("CPU request '1'", r.stderr) + finally: + box.close() + + def test_l_stage_drops_only_the_download_and_the_done_conversion(self): + r = self.box.render("L", os.path.join(self.out, "L")) + self.assertEqual(r.returncode, 0, r.stderr) + raw, staged = self.docs("L", "agent.yaml"), self.docs("L") + self.assertEqual(sorted(d["kind"] for d in raw), + ["ConfigMap", "Deployment", "Job", "Namespace", "PersistentVolumeClaim", "Service"]) + self.assertEqual(sorted(d["kind"] for d in staged), ["Deployment", "Namespace", "Service"]) + rc, sc = (run.deployment(x)["spec"]["template"]["spec"]["containers"][0] for x in (raw, staged)) + ra = [str(x) for x in rc["args"]] + i = ra.index("--model-url") + self.assertEqual(sc["args"], ra[:i] + ra[i + 2:]) + self.assertEqual((sc["image"], sc["resources"]), (rc["image"], rc["resources"])) + a = sc["args"] + self.assertEqual(run.value(a, "--model"), run.GGUF_PATH) + self.assertEqual(run.value(a, "--spec-draft-model"), run.DRAFT_PATH) + self.assertEqual(run.value(a, "--spec-type"), "draft-mtp") + self.assertEqual((run.value(a, "--threads"), run.value(a, "--threads-batch")), ("8", "16")) + self.assertTrue(all(m["readOnly"] for m in sc["volumeMounts"] if m["name"] == "models")) + vol = run.deployment(staged)["spec"]["template"]["spec"]["volumes"] + self.assertEqual(vol, [{"name": "models", "hostPath": {"path": "/var/apb-models", "type": "Directory"}}]) + self.assertNotIn("initContainers", run.deployment(staged)["spec"]["template"]["spec"]) + fit = jl(os.path.join(self.out, "L", "fit.json")) + self.assertEqual((fit["threads"], fit["threads-batch"]), ("8", "16")) + self.assertTrue(fit["fit"].startswith("cpu ")) + + def test_o_stage_keeps_the_pull_container(self): + r = self.box.render("O", os.path.join(self.out, "O")) + self.assertEqual(r.returncode, 0, r.stderr) + raw, staged = self.docs("O", "agent.yaml"), self.docs("O") + rs, ss = (run.deployment(x)["spec"]["template"]["spec"] for x in (raw, staged)) + self.assertEqual(ss["initContainers"], rs["initContainers"]) + self.assertEqual(ss["containers"], rs["containers"]) + self.assertEqual(ss["volumes"], [{"name": "models", "hostPath": {"path": "/var/apb-models", "type": "Directory"}}]) + fit = jl(os.path.join(self.out, "O", "fit.json")) + self.assertIsNotNone(fit["max_num_seqs"]) + self.assertFalse(any(x.startswith("--draft") for x in fit["pull_args"])) + + def test_render_needs_prep_s_drafter_job(self): + os.remove(os.path.join(self.box.g, "models", run.DRAFT_DIR, "prep-job.json")) + r = self.box.render("L", os.path.join(self.out, "L")) + self.assertNotEqual(r.returncode, 0) + self.assertIn("no prep-job.json", r.stderr) + put(os.path.join(self.box.g, "models", run.DRAFT_DIR, "prep-job.json"), json.dumps(dict(self.job, image="other"))) + r = self.box.render("L", os.path.join(self.out, "L2")) + self.assertIn("differs from the one prep.sh ran", r.stderr) + + def test_prep_job_is_pr182s_conversion(self): + self.assertEqual(self.job["image"], run.LLAMA_FULL_IMAGE) + self.assertEqual(self.job["args"], [run.DRAFT_REPO, run.DRAFT_REV, "/models/" + run.DRAFT_DIR, "4294967296", + "Q4_0", "--outtype", "bf16"]) + self.assertIn("transformers==5.21.0", self.job["env"]["GGUF_CONVERT_PIP"]) + + def test_check_names_each_deviation(self): + for arm in ("O", "L"): + self.assertEqual(self.box.render(arm, os.path.join(self.out, arm)).returncode, 0) + base = {arm: self.docs(arm) for arm in ("O", "L")} + + def bad(arm, edit): + docs = json.loads(json.dumps(base[arm])) + edit(run.deployment(docs)["spec"]["template"]) + return run.check(arm, "desktop", docs) + + def c(t): + return t["spec"]["containers"][0] + self.assertEqual(run.check("O", "desktop", base["O"]), []) + self.assertEqual(run.check("L", "desktop", base["L"]), []) + self.assertTrue(bad("L", lambda t: c(t).update(image="ghcr.io/ggml-org/llama.cpp:server-b11749"))) + self.assertTrue(bad("L", lambda t: c(t)["args"].remove("--spec-type"))) + self.assertTrue(bad("L", lambda t: c(t)["args"].extend(["--model-url", "https://x"]))) + self.assertTrue(bad("L", lambda t: c(t)["args"].__setitem__(c(t)["args"].index("--threads") + 1, "16"))) + self.assertTrue(bad("L", lambda t: [m.pop("readOnly") for m in c(t)["volumeMounts"]])) + self.assertTrue(bad("O", lambda t: t["spec"]["initContainers"][0]["args"].append("--draft_source_model=x"))) + self.assertTrue(bad("O", lambda t: t["metadata"]["labels"].pop("k3d-gpu.io/cores"))) + self.assertTrue(bad("O", lambda t: c(t)["resources"]["requests"].update(cpu="1"))) + self.assertTrue(bad("O", lambda t: c(t)["resources"]["limits"].update({"gpu.intel.com/xe": 1}))) + self.assertTrue(bad("O", lambda t: t["spec"]["initContainers"][0]["args"].remove("--target_device=CPU"))) + # On crescent no hybrid balloon is expected: the P-core label is a deviation there. + self.assertIn("on a host with no hybrid balloon", " ".join(run.check("O", "crescent", base["O"]))) + + +class Crescent(unittest.TestCase): + def test_xeon_render_has_no_balloon(self): + box = Box("crescent", cluster="k3dgpu-g-h59-1") + try: + box.prep_job() + for arm in ("O", "L"): + r = box.render(arm, os.path.join(box.d, arm)) + self.assertEqual(r.returncode, 0, r.stderr) + tpl = run.deployment(run.load_docs(os.path.join(box.d, arm, "deploy.yaml")))["spec"]["template"] + self.assertNotIn("k3d-gpu.io/cores", tpl["metadata"]["labels"]) + finally: + box.close() + + +def phases_line(**k): + return json.dumps(k) + "\n" + + +class Gate(unittest.TestCase): + def setUp(self): + self.d = tempfile.mkdtemp() + + def tearDown(self): + shutil.rmtree(self.d) + + def arm(self, hold, arm, tries, status="measured"): + ad = os.path.join(self.d, f"hold{hold}", arm) + os.makedirs(ad, exist_ok=True) + put(os.path.join(ad, "status.json"), json.dumps({"status": status})) + with open(os.path.join(ad, "phases.jsonl"), "w") as f: + f.write(phases_line(phase="n1", t0=0, t1=1, check={"void": True, "reason": "n1 does not decide"})) + for i, v in enumerate(tries): + f.write(phases_line(phase="n4", **{"try": i + 1}, t0=1, t1=2, check={"void": v, "reason": f"cpu {v}"})) + + def test_matched_when_every_n4_ends_clean(self): + for k in (1, 2, 3): + self.arm(k, "O", [False]) + self.arm(k, "L", [True, False]) + g = run.gate(self.d) + self.assertTrue(g["matched"], g) + self.assertEqual(g["checked"], 6) + + def test_void_twice_is_unmatched(self): + for k in (1, 2, 3): + self.arm(k, "O", [False]) + self.arm(k, "L", [False]) + self.arm(2, "L", [True, True]) + g = run.gate(self.d) + self.assertFalse(g["matched"]) + self.assertIn("hold 2 L", g["reason"]) + + def test_spare_stands_in_and_unmeasured_arms_are_skipped(self): + for k in (1, 2, 3): + self.arm(k, "O", [False]) + self.arm(k, "L", [False]) + self.arm(3, "L", [True, True]) + self.arm(5, "O", [False]) + self.arm(5, "L", [False]) + put(os.path.join(self.d, "hold5", "repeats"), "3\n") + self.assertTrue(run.gate(self.d)["matched"]) + self.arm(1, "O", [True, True], status="failed") + self.assertTrue(run.gate(self.d)["matched"]) + + def test_missing_check_is_unmatched(self): + for k in (1, 2, 3): + self.arm(k, "O", [False]) + self.arm(k, "L", [False]) + with open(os.path.join(self.d, "hold1", "O", "phases.jsonl"), "w") as f: + f.write(phases_line(phase="n4", t0=1, t1=2)) + self.assertFalse(run.gate(self.d)["matched"]) + + +class Prefix(unittest.TestCase): + def test_six_agents_one_shared_prefix(self): + srv = serve({"v1": 0.0, "v3": 0.0}) + out = os.path.join(tempfile.mkdtemp(), "prefix.json") + try: + rec = run.prefix(f"http://127.0.0.1:{srv.server_address[1]}/v1", "gemma", 5901, out) + finally: + srv.shutdown() + srv.server_close() + bodies = [b for _, b in srv.bodies] + self.assertEqual(len(bodies), 6) + self.assertEqual(len({json.dumps(b["messages"][0]) for b in bodies}), 1) + self.assertEqual(len({json.dumps(b["tools"]) for b in bodies}), 1) + self.assertEqual(len({b["messages"][1]["content"] for b in bodies}), 6) + for b in bodies: + self.assertEqual((b["model"], b["max_tokens"], b["stream"]), ("gemma", 8, True)) + self.assertEqual(b["chat_template_kwargs"], {"enable_thinking": False}) + self.assertEqual([a["agent"] for a in rec["agents"]], ["a1", "a2", "a3", "a4", "a5", "a6"]) + self.assertIsNotNone(rec["added_ttft_median_s"]) + self.assertEqual(jl(out)["seed"], 5901) + + +def run_arm(box, arm, hold_dir, mode, holdnum, port, **extra): + env = dict(box.env, MODE=mode, DEADLINE=str(int(time.time()) + 5000), HOLDNUM=str(holdnum), H59_PORT=str(port)) + env.update(extra) + r = subprocess.run(["bash", os.path.join(HERE, "arm.sh"), arm, os.path.join(hold_dir, arm)], env=env, + capture_output=True, text=True, timeout=600) + return r + + +def synth_load(hold_dir, arms, foreign=1.0): + """H57d's sampler records over the hold's phases: P-core busy and node CPU, matched.""" + t = [] + for a in arms: + for line in rd(os.path.join(hold_dir, a, "phases.jsonl")).splitlines(): + r = json.loads(line) + t += [r["t0"], r["t1"]] + with open(os.path.join(hold_dir, "load.jsonl"), "w") as f: + ts = min(t) - 1 + while ts < max(t) + 1: + f.write(json.dumps({"ts": ts, "seconds": 0.2, "pcore_busy": 6.0 + foreign, "node_cpu": 6.0}) + "\n") + ts += 0.2 + + +class EndToEnd(unittest.TestCase): + """arm.sh, as hold-run.sh runs it, for the registered holds of a desktop row: three + performance holds in their orders and the quality hold; then h59.py scores the records.""" + + def test_holds_feed_the_scorer(self): + box = Box("desktop") + srv = serve({"v1": 0.0008, "v3": 0.0016}) # L decodes about twice as fast as O + try: + box.prep_job() + res = os.path.join(box.g, "results", "H59d") + for k in (1, 2, 3, 4): + mode, arms = run.hold(str(k)) + hd = os.path.join(res, f"hold{k}") + for a in arms: + r = run_arm(box, a, hd, mode, k, srv.server_address[1], CLUSTER=box.cluster) + st = jl(os.path.join(hd, a, "status.json")) + self.assertEqual(st["status"], "measured", f"hold {k} {a}: {st} {r.stderr[-2000:]}") + if mode == "perf": + synth_load(hd, arms) + for k in (1, 2, 3): + ad = os.path.join(res, f"hold{k}", "L") + for f in ("load-n1.json", "load-n4.json", "load-ttft.json", "prefix.json", "cpus.txt", "fit.json", + "deploy.yaml", "energy-n4.json", "energy-idle.json"): + self.assertTrue(os.path.exists(os.path.join(ad, f)), f"hold {k} L {f}") + self.assertFalse(os.path.exists(os.path.join(ad, "load-n16.json"))) + n4 = jl(os.path.join(ad, "load-n4.json"))["phases"][0] + self.assertEqual((n4["concurrency"], len(n4["samples"])), (4, 16)) + self.assertEqual([json.loads(x)["phase"] for x in rd(os.path.join(ad, "phases.jsonl")).splitlines()], + ["idle", "n1", "n4", "ttft", "prefix"]) + q = jl(os.path.join(res, "hold4", "O", "quality.json")) + self.assertEqual((q["n"], q["errors"]), (50, 0)) + rep = h59.score(res, "h57d") + self.assertEqual(rep["verdict"], "SUPPORTED", json.dumps({k: v for k, v in rep.items() if k != "arms"})) + self.assertEqual([rep["arms"][a]["n4_requests"] for a in ("L", "O")], [48, 48]) + self.assertGreater(rep["ci"][0], 1.2) + self.assertTrue(rep["all_holds_faster"]) + self.assertTrue(rep["load_gate"]["matched"]) + # The scorer's CLI on the same records. + cli = subprocess.run([sys.executable, os.path.join(HERE, "h59.py"), "score", res, "--load", "h57d"], + capture_output=True, text=True) + self.assertEqual(json.loads(cli.stdout)["verdict"], "SUPPORTED") + finally: + srv.shutdown() + srv.server_close() + box.close() + + def test_crescent_n16_and_the_void_rerun(self): + box = Box("crescent", cluster="k3dgpu-g-h59-1") + srv = serve({"v1": 0.0004, "v3": 0.0004}) + try: + box.prep_job() + serving = os.path.join(box.d, "serving.jsonl") + now = time.time() + with open(serving, "w") as f: # non-campaign CPU 9 CPU-s/s throughout: every phase is void + for i in range(-20, 900): + f.write(json.dumps({"ts": now + i, "seconds": 1, "serving_cpu": 9.0, "aiperf_procs": 0}) + "\n") + hd = os.path.join(box.g, "results", "H59", "hold1") + r = run_arm(box, "L", hd, "perf", 1, srv.server_address[1], SERVING=serving) + st = jl(os.path.join(hd, "L", "status.json")) + self.assertEqual(st["status"], "measured", r.stderr[-2000:]) + ph = [json.loads(x) for x in rd(os.path.join(hd, "L", "phases.jsonl")).splitlines()] + self.assertEqual([p["phase"] for p in ph if p["phase"] == "n4"], ["n4", "n4"]) + self.assertTrue(all(p["check"]["void"] for p in ph if p["phase"] == "n4")) + for f in ("load-n16.json", "load-n4-void1.json", "energy-idle-n4.json"): + self.assertTrue(os.path.exists(os.path.join(hd, "L", f)), f) + n16 = jl(os.path.join(hd, "L", "load-n16.json"))["phases"][0] + self.assertEqual((n16["concurrency"], len(n16["samples"])), (16, 32)) + g = run.gate(os.path.join(box.g, "results", "H59")) + self.assertFalse(g["matched"]) + self.assertIn("hold 1 L", g["reason"]) + finally: + srv.shutdown() + srv.server_close() + box.close() + + def test_render_refusal_and_not_reached(self): + box = Box("desktop", stub_cluster="elsewhere") + try: + box.prep_job() + hd = os.path.join(box.d, "hold1") + run_arm(box, "O", hd, "perf", 1, 1) + st = jl(os.path.join(hd, "O", "status.json")) + self.assertEqual(st["status"], "failed") + self.assertIn("render not the registered arm", st["reason"]) + self.assertFalse(os.path.exists(os.path.join(box.d, "stub", "applied.yaml"))) + r = run_arm(box, "O", os.path.join(box.d, "hold2"), "perf", 2, 1, DEADLINE=str(int(time.time()) + 100)) + st = jl(os.path.join(box.d, "hold2", "O", "status.json")) + self.assertEqual(st["status"], "not_reached", r.stderr) + finally: + box.close() + + +class HoldGuards(unittest.TestCase): + """hold.sh refuses, before the lock, what would spend a hold on the wrong thing.""" + + def setUp(self): + self.d = tempfile.mkdtemp() + self.g = os.path.join(self.d, "g") + put(os.path.join(self.d, "cpuinfo"), "processor\t: 0\nmodel name\t: 13th Gen Intel(R) Core(TM) i9-13900T\n") + os.makedirs(os.path.join(self.g, "bin")) + shutil.copy(os.path.join(HERE, "k3d-shim.sh"), os.path.join(self.g, "bin", "k3d")) + m = os.path.join(self.g, "models") + for p in run.models_ready(self.g): + put(p, "x") + self.assertEqual(run.models_ready(self.g), []) + # flock stands in so a hold that passes every guard stops at the lock (exit 4). + put(os.path.join(self.d, "bin", "flock"), "#!/bin/sh\nexit 1\n", 0o755) + self.env = {"PATH": os.path.join(self.d, "bin") + ":/usr/bin:/bin", "HOME": self.d, "G": self.g, + "H59_IMAGE": NODE_IMAGE, "H59_CPUINFO": os.path.join(self.d, "cpuinfo"), + "LOCK": os.path.join(self.d, "lock")} + self.res = os.path.join(self.g, "results", "H59d") + self.m = m + + def tearDown(self): + shutil.rmtree(self.d) + + def hold(self, *args, **env): + return subprocess.run(["bash", os.path.join(HERE, "hold.sh"), *args], env=dict(self.env, **env), + capture_output=True, text=True, timeout=60) + + def ran(self, k, statuses): + hd = os.path.join(self.res, f"hold{k}") + put(os.path.join(hd, "rc"), "0\n") + for a, s in statuses.items(): + put(os.path.join(hd, a, "status.json"), json.dumps({"status": s})) + + def test_guards_pass_to_the_lock(self): + r = self.hold("desktop", "1") + self.assertEqual(r.returncode, 4, r.stderr) + + def test_wrong_cpu_class(self): + r = self.hold("wildct", "1") + self.assertEqual(r.returncode, 2) + self.assertIn("not the i7-13700", r.stderr) + + def test_unregistered_holds(self): + for args in (("desktop", "7"), ("desktop", "5"), ("desktop", "01"), ("desktop", "1", "2"), ("nowhere", "1")): + self.assertEqual(self.hold(*args).returncode, 2, args) + + def test_image_pinned_by_digest(self): + self.assertEqual(self.hold("desktop", "1", H59_IMAGE="cryptoandcoffee/k3d-gpu:0.1.80").returncode, 2) + + def test_shim_and_models(self): + os.remove(os.path.join(self.g, "bin", "k3d")) + self.assertIn("k3d-shim.sh", self.hold("desktop", "1").stderr) + shutil.copy(os.path.join(HERE, "k3d-shim.sh"), os.path.join(self.g, "bin", "k3d")) + os.remove(os.path.join(self.m, run.DRAFT_DIR, "prep-job.json")) + r = self.hold("desktop", "1") + self.assertEqual(r.returncode, 2) + self.assertIn("prep.sh", r.stderr) + + def test_a_hold_runs_once(self): + self.ran(2, {"L": "measured", "O": "failed"}) + r = self.hold("desktop", "2") + self.assertEqual(r.returncode, 2) + self.assertIn("a failed hold counts", r.stderr) + + def test_spares_repeat_a_failed_hold_only(self): + self.assertEqual(self.hold("desktop", "5", "2").returncode, 2) # hold 2 has not run + self.ran(2, {"L": "measured", "O": "measured"}) + self.assertEqual(self.hold("desktop", "5", "2").returncode, 2) # nothing failed in it + self.ran(3, {"O": "measured", "L": "not_deliverable"}) + self.assertEqual(self.hold("desktop", "5", "3").returncode, 4) # passes to the lock + self.assertFalse(os.path.exists(os.path.join(self.res, "hold5", "repeats"))) + + def test_node_image_is_the_rows(self): + put(os.path.join(self.res, "node-image"), "cryptoandcoffee/k3d-gpu:0.1.79@sha256:" + "b" * 64 + "\n") + r = self.hold("desktop", "1") + self.assertEqual(r.returncode, 2) + self.assertIn("node image", r.stderr) + + +class Shim(unittest.TestCase): + def test_volume_only_on_this_campaigns_clusters(self): + d = tempfile.mkdtemp() + try: + put(os.path.join(d, "k3d"), "#!/bin/sh\necho \"$@\"\n", 0o755) + env = {"PATH": "/usr/bin:/bin", "K3D_REAL": os.path.join(d, "k3d"), "H59_G": "/g", "H59_PREFIX": "k3dgpu-d-h59-"} + shim = os.path.join(HERE, "k3d-shim.sh") + out = subprocess.run(["bash", shim, "cluster", "create", "k3dgpu-d-h59-1", "--x"], env=env, + capture_output=True, text=True).stdout + self.assertEqual(out.strip(), "cluster create k3dgpu-d-h59-1 --x --volume=/g/models:/var/apb-models@server:*") + out = subprocess.run(["bash", shim, "cluster", "create", "k3dgpu-d-h57d-1"], env=env, capture_output=True, text=True).stdout + self.assertEqual(out.strip(), "cluster create k3dgpu-d-h57d-1") + finally: + shutil.rmtree(d) + + +if __name__ == "__main__": + unittest.main() diff --git a/bench/wildct/row.sh b/bench/wildct/row.sh index 62415cf9..62d16aba 100755 --- a/bench/wildct/row.sh +++ b/bench/wildct/row.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# row.sh ROW [HOLD MODE [ARM...]]: one wildct row's command as the wildct pre-registration lists it ("Per +# row.sh ROW [ARGS...]: one wildct row's command as the wildct pre-registration lists it ("Per # row") with amendment R1, run as bench/wildct/hold.sh's CMD (hold.sh exports # K3D_GPU_RAPL_EXEC, rule W4, and WILDCT_HOLD, the hold's name) from the shipped repository # archive. The release's node image and the models are staged first (bench/wildct/RESUME.md). @@ -19,6 +19,11 @@ # h57w-convert H57d's bench/h57d/convert.sh (drafter GGUF, Od's graph.pbtxt) under the host # lock, not in a hold; at most 3600 s # h57wb H57wb's start reading (/ available, MemAvailable) and its decision; not a hold +# h59w HOLD [REPEATS] H59w: bench/h59/hold.sh wildct (inner lock; H59 amendment R1): holds 1-3 +# perf and 4 quality in their registered orders; a spare 5 or 6 names the failed +# hold it repeats +# h59w-prep bench/h59/prep.sh wildct: the downloads, then the drafter conversion under the +# host lock; not in a hold # Env: IMAGE the release node image as name:tag@sha256:digest (every row but h52w and h57wb) # OUT output directory (default ~/k3dgpu-bench-wildct/runs/${WILDCT_HOLD:-ROW}/out) # WILDCT_ROW_PRINT=1 print the environment (one NAME=value per line) and the command, run nothing @@ -95,6 +100,21 @@ case ${row} in ARGV=(timeout -k 30 3600 bash "${REPO}/bench/h57d/convert.sh") ;; h57wb) ARGV=(h57wb) ;; + h59w) + need_image + case ${2:-}:${3:-}:$# in + [1234]::2 | [56]:[1234]:3) ;; + *) echo "row.sh: h59w ${*:2}: holds 1-3 perf and 4 quality take no other argument; a spare 5 or 6 names the hold it repeats" >&2; exit 2 ;; + esac + E=(G="${W}/h59" LOCK="${W}/inner.lock" K3D_REAL="${W}/bin/k3d" H59_IMAGE="${IMAGE}") + ARGV=(bash "${REPO}/bench/h59/hold.sh" wildct "${@:2}") + [ "${WILDCT_ROW_PRINT:-0}" = 1 ] || cmp -s "${REPO}/bench/h59/k3d-shim.sh" "${W}/h59/bin/k3d" \ + || { echo "row.sh: ${W}/h59/bin/k3d is not bench/h59/k3d-shim.sh (install it first)" >&2; exit 2; } ;; + h59w-prep) + # prep.sh takes the host lock itself for the conversion, so it never runs inside hold.sh. + [ -z "${WILDCT_HOLD:-}" ] || { echo "row.sh: h59w-prep runs outside hold.sh" >&2; exit 2; } + E=(G="${W}/h59" LOCK="${W}/live.lock" PY_HF="${W}/hfvenv/bin/python") + ARGV=(bash "${REPO}/bench/h59/prep.sh" wildct) ;; *) echo "row.sh: unknown row ${row}" >&2; exit 2 ;; esac [ "${WILDCT_ROW_CHECK:-0}" != 1 ] || exit 0 diff --git a/tests/wildct_hold_test.sh b/tests/wildct_hold_test.sh index 90edff80..96e779a1 100755 --- a/tests/wildct_hold_test.sh +++ b/tests/wildct_hold_test.sh @@ -608,6 +608,19 @@ row_envs_match_the_registration() { "bash ${ROOT}/bench/h57d/hold.sh 2 perf" || return 1 has h57w 5 perf Ld Ld-mtp Od -- "bash ${ROOT}/bench/h57d/hold.sh 5 perf Ld Ld-mtp Od" || return 1 has h57w-convert -- "G=${HOME}/k3dgpu-bench-wildct/h57" "LOCK=${HOME}/k3dgpu-bench-wildct/live.lock" || return 1 + # H59w (H59 amendment R1): bench/h59/hold.sh's wildct profile carries the opt-out, W3 and load1 12. + has h59w 2 -- "G=${HOME}/k3dgpu-bench-wildct/h59" "LOCK=${HOME}/k3dgpu-bench-wildct/inner.lock" \ + "K3D_REAL=${HOME}/k3dgpu-bench-wildct/bin/k3d" "H59_IMAGE=${IMG}" "bash ${ROOT}/bench/h59/hold.sh wildct 2" || return 1 + has h59w 5 3 -- "bash ${ROOT}/bench/h59/hold.sh wildct 5 3" || return 1 + has h59w-prep -- "G=${HOME}/k3dgpu-bench-wildct/h59" "LOCK=${HOME}/k3dgpu-bench-wildct/live.lock" \ + "bash ${ROOT}/bench/h59/prep.sh wildct" || return 1 + python3 -c 'import sys +sys.path.insert(0, sys.argv[1] + "/bench/h59") +import run +p = run.PROFILES["wildct"] +sys.exit(not (p["k3s_args"] == sys.argv[2] and p["vendors"] == "auto,-nvidia,-intel" and p["load1_max"] == 12 + and p["prefix"].startswith("k3dgpu-w-")))' "${ROOT}" "${W3_LEDGER}" \ + || { echo "bench/h59/run.py's wildct profile is not the registration's"; return 1; } has h18w -- "bash ${ROOT}/bench/sandbox_round.sh (runs 1-3)" || return 1 # H52w's procedure: `hold.sh H52w 1800 3 bench/wildct/h52w.sh `; h52w.sh sets its own # cluster, opt-out and W3, and runs the installed release's default node image (no IMAGE). @@ -687,6 +700,8 @@ row_guards() { IMAGE=cryptoandcoffee/k3d-gpu:0.1.74 bash "${ROW}" h25w 2> /dev/null; [ $? = 2 ] || { echo "IMAGE without a digest accepted"; return 1; } IMAGE=${IMG} bash "${ROW}" h57w 1 perf 2> /dev/null; [ $? = 2 ] || { echo "h57w ran without the k3d shim"; return 1; } WILDCT_HOLD=X bash "${ROW}" h57w-convert 2> /dev/null; [ $? = 2 ] || { echo "h57w-convert ran inside a hold"; return 1; } + IMAGE=${IMG} bash "${ROW}" h59w 1 2> /dev/null; [ $? = 2 ] || { echo "h59w ran without the k3d shim"; return 1; } + WILDCT_HOLD=X bash "${ROW}" h59w-prep 2> /dev/null; [ $? = 2 ] || { echo "h59w-prep ran inside a hold"; return 1; } # WILDCT_ROW_CHECK=1 (launch's pre-check): every check, the k3d shim included; runs nothing. local a chk() { WILDCT_ROW_CHECK=1 IMAGE=${IMG} bash "${ROW}" "$@" > "${W}/chk.out" 2>&1; } @@ -701,6 +716,16 @@ row_guards() { # shellcheck disable=SC2086 # the row's arguments, split on purpose chk h57w ${a}; [ $? = 2 ] || { echo "check mode passed h57w ${a}"; return 1; } done + chk h59w 1; [ $? = 2 ] || { echo "check mode passed h59w without the k3d shim"; return 1; } + install -D -m 0755 "${ROOT}/bench/h59/k3d-shim.sh" "${HOME}/k3dgpu-bench-wildct/h59/bin/k3d" + for a in 1 2 3 4 "5 1" "6 4"; do + # shellcheck disable=SC2086 # the row's arguments, split on purpose + chk h59w ${a} || { echo "check mode refused h59w ${a}: $(cat "${W}/chk.out")"; return 1; } + done + for a in "" 5 7 0 "1 2" "5 5" "6 0" "1 perf" "5 1 O"; do + # shellcheck disable=SC2086 # the row's arguments, split on purpose + chk h59w ${a}; [ $? = 2 ] || { echo "check mode passed h59w ${a}"; return 1; } + done chk h25w || { echo "check mode refused h25w: $(cat "${W}/chk.out")"; return 1; } [ "$(calls '')" = 0 ] || { echo "check mode ran a command: $(cat "${W}/calls.log")"; return 1; } bash "${ROW}" nope 2> /dev/null; [ $? = 2 ] || { echo "unknown row accepted"; return 1; } From 993605d9ea71554b022e91f991c6cdd98627d8b2 Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Thu, 29 Oct 2026 22:51:28 -0700 Subject: [PATCH 2/2] bench(h59): review fixes to amendment R1: CPU class, spares, interrupted holds, pinned test images The images.lock check failed on three tag-only image strings in bench/h59/test_run.py. They are now derived from the pinned references in run.py and the test's node image, so no unpinned literal is left in the file. hold.sh compared the host's CPU by substring, so an i7-13700K passed as the i7-13700 and an i9-13900TE as the i9-13900T. The engine rule matches a row by the exact model number (console/model_opt.py class_tokens), so the host check now uses that function (run.py cpu). A spare could repeat a hold that spare 5 had already measured, and h59.py reads spare 6 over spare 5, so a measured record could be replaced. Spare 6 now runs only after spare 5, and a spare repeats a hold only while the record standing for it has an arm not measured. A hold interrupted before it recorded its rc could run again under the same number. hold.sh now writes `started` once the floors pass, refuses a hold that has one, and spare-ok counts it as run. test_run.py now checks the server's request log against the registered counts and settings (3 warmup, 8 at N=1, 16 at N=4, 4 TTFT requests at 16 tokens on H54's prompt, 6 prefix agents, GSM8K 0-49 at 1024 tokens with thinking off, unique prompts, the 900 s boot limit). Before this a runner with 6 N=1 requests, 32-token TTFT requests or a 600 s boot limit passed every test. The new guard tests failed on the previous commit. Amendment R1 records the spare order, the `started` file and the CPU class rule, and cites OVMS's `--task` option at src/cli_parser.cpp:324, where it is defined (:62 reads it). No prediction, statistic, gate, cap or hold count changes. --- EXPERIMENTS.md | 24 +++++++++----- bench/h59/hold.sh | 24 +++++++++----- bench/h59/run.py | 59 +++++++++++++++++++++++++++------ bench/h59/test_run.py | 77 +++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 153 insertions(+), 31 deletions(-) diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 75b82202..d20d0dd3 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -12799,8 +12799,9 @@ Tests: `bench/h59/test_run.py` and the `h59w` rows of `tests/wildct_hold_test.sh render both arms through the real `k3d-gpu agent manifest` against the fit fixtures, with a kubectl stand-in that answers only for the hold's own context. They run `arm.sh` against a local OpenAI-compatible server and score its records with `h59.py`. Four holds of a desktop row feed -the bootstrap-ratio scorer end to end. A crescent arm runs its N=16 phase and re-runs a void N=4 -phase. +the bootstrap-ratio scorer end to end, and the server's request log is checked against the +registered counts and settings (warmup, N=1, N=4, TTFT, prefix, GSM8K, the 900 s boot limit). A +crescent arm runs its N=16 phase and re-runs a void N=4 phase. **Arms as rendered.** Each arm is `scripts/k3d-gpu agent manifest --model ` at the runner's commit, run in the hold's cluster. O is `--model google/gemma-4-26B-A4B-it`. L is `--model @@ -12856,7 +12857,7 @@ none is dropped. `--spec-type` :4330. `draft-mtp` is a type in `common/speculative.cpp` :78. - Defaults L relies on: the prompt cache is on (`common/common.h` :639, `cache_prompt = true`), and the draft length is 3 (:331). -- OVMS `v2026.4.1`, `src/cli_parser.cpp`: `--task` :62, `--rest_port` :93, `--log_level` :117, +- OVMS `v2026.4.1`, `src/cli_parser.cpp`: `--task` :324, `--rest_port` :93, `--log_level` :117, `--cache_dir` :152, `--pull` :236, `--source_model` :240, `--model_repository_path` :256, `--model_name` :270, `--model_path` :274 and `--target_device` :314. - OVMS `src/graph_export/graph_cli_parser.cpp`: `--max_num_seqs` :44, `--enable_prefix_caching` @@ -12906,12 +12907,17 @@ writes `graph.pbtxt`. Weights are never copied into the node. - Holds 1-3 are performance holds in the orders (O, L), (L, O), (O, L). Hold 4 is the quality hold, in (L, O). -- A spare 5 or 6 repeats hold REPEATS in its order. It may only repeat a hold that ran and left - an arm not measured. It writes the file `repeats`, which `h59.py` reads, only once it runs. -- A hold runs once. A hold that recorded its `rc` is refused, so a failed hold counts and only a - spare repeats it. There is no hold number above 6. -- Before the lock, `hold.sh` refuses a host whose CPU model is not the row's, a node image not - pinned by digest, a `${G}/bin/k3d` that is not the shim, and missing model files. +- A spare 5 or 6 repeats hold REPEATS in its order. It may only repeat a hold that ran and whose + standing record (the hold, or spare 5 once spare 5 repeated it) left an arm not measured. + Spare 6 runs only after spare 5, since `h59.py` reads spare 6 over spare 5 for the same hold. + A spare writes the file `repeats`, which `h59.py` reads, only once it runs. +- A hold runs once. Once the floors pass, `hold.sh` writes the hold's `started` file, before + anything runs. A hold with a `started` file or an `rc` is refused, so a failed or interrupted + hold counts and only a spare repeats it. There is no hold number above 6. +- Before the lock, `hold.sh` refuses a host whose CPU class is not the row's, a node image not + pinned by digest, a `${G}/bin/k3d` that is not the shim, and missing model files. The CPU + class is the model number the optimizer matches a row by (`console/model_opt.py` + `class_tokens` of the first `model name`), compared exactly: an i7-13700K is not an i7-13700. - The node image (`H59_IMAGE`) is the k3d-gpu release whose launcher is this commit's or later, by digest. The row's first hold records it, and later holds must run the same image. - A wiring smoke (`hold.sh HOST smoke`) runs hold 1's arms with one 16-token request and one diff --git a/bench/h59/hold.sh b/bench/h59/hold.sh index 6767ce3a..af373cff 100644 --- a/bench/h59/hold.sh +++ b/bench/h59/hold.sh @@ -3,12 +3,15 @@ # amendment R1") on HOST = crescent (H59, Xeon 6741P), desktop (H59d, i9-13900T) or wildct (H59w, # i7-13700, run as bench/wildct/hold.sh's command: bench/wildct/row.sh h59w). HOLD: 1-3 perf in # the orders (O, L), (L, O), (O, L); 4 quality in (L, O); a spare 5 or 6 repeats hold REPEATS -# (one that ran and left an arm not measured) in its order; smoke runs hold 1's arms with no -# scored request (discarded, disclosed). Each hold runs once: a failed hold counts. +# in its order, when the record standing for it (the hold, or spare 5's repeat of it) left an +# arm not measured; spare 6 runs only after spare 5. Smoke runs hold 1's arms with no scored +# request (discarded, disclosed). Each hold runs once: a hold that started (its `started` file, +# written once the floors pass) counts, failed or interrupted. # -# Before the lock it refuses (exit 2) a host whose CPU is not the profile's, a node image not -# pinned by digest or not the one the row's first hold ran, a ${G}/bin/k3d that is not -# bench/h59/k3d-shim.sh, and models prep.sh has not left in ${G}/models. Then the host's own rules +# Before the lock it refuses (exit 2) a host whose CPU class is not the profile's (the model +# number the optimizer matches, exactly: run.py cpu), a node image not pinned by digest or not +# the one the row's first hold ran, a ${G}/bin/k3d that is not bench/h59/k3d-shim.sh, and models +# prep.sh has not left in ${G}/models. Then the host's own rules # (run.py profile): its lock (flock -w 21600; crescent ~/k3dgpu-bench.lock, else LOCK); the # hold-start rule (load1 <= load1_max, and on crescent no aiperf process; 2-min polls, at most 60 # min, then the hold runs flagged); the floors (crescent: / at most 97% used; desktop and wildct: @@ -34,17 +37,18 @@ G=${G/#\~/${HOME}} order=$(python3 "${RUN}" hold "${HOLD}" ${REPEATS:+"${REPEATS}"}) || exit 2 read -r MODE ARMS_RUN <<< "${order}" read -r -a ARMS_RUN <<< "${ARMS_RUN}" -grep -m1 '^model name' "${H59_CPUINFO:-/proc/cpuinfo}" | grep -qF -- "${CPU}" \ - || { echo "hold.sh: this host's CPU is not the ${CPU} that ${ROW} registers" >&2; exit 2; } +got=$(python3 "${RUN}" cpu "${HOST}" "${H59_CPUINFO:-/proc/cpuinfo}") \ + || { echo "hold.sh: this host's CPU class is ${got:-unknown}, not the ${CPU} that ${ROW} registers" >&2; exit 2; } [[ "${H59_IMAGE:-}" =~ ^[^@]+:[^@]+@sha256:[0-9a-f]{64}$ ]] \ || { echo "hold.sh: set H59_IMAGE to the k3d-gpu node image as name:tag@sha256:" >&2; exit 2; } cmp -s "${HERE}/k3d-shim.sh" "${G}/bin/k3d" || { echo "hold.sh: ${G}/bin/k3d is not bench/h59/k3d-shim.sh (install it first)" >&2; exit 2; } python3 "${RUN}" models "${G}" >&2 || { echo "hold.sh: run bench/h59/prep.sh ${HOST} first" >&2; exit 2; } RES=${G}/results/${ROW} case ${HOLD} in smoke) R=${G}/smoke/${ROW}-$(date -u +%Y%m%dT%H%M%SZ) ;; *) R=${RES}/hold${HOLD} ;; esac -[ ! -e "${R}/rc" ] || { echo "hold.sh: hold ${HOLD} has run (${R}/rc); a failed hold counts, a spare repeats it" >&2; exit 2; } +[ ! -e "${R}/rc" ] && [ ! -e "${R}/started" ] \ + || { echo "hold.sh: hold ${HOLD} has run (${R}); a failed hold counts, interrupted or not, and a spare repeats it" >&2; exit 2; } if [ -n "${REPEATS}" ]; then - why=$(python3 "${RUN}" spare-ok "${RES}" "${REPEATS}") || { echo "hold.sh: spare ${HOLD} cannot repeat hold ${REPEATS}: ${why}" >&2; exit 2; } + why=$(python3 "${RUN}" spare-ok "${RES}" "${REPEATS}" "${HOLD}") || { echo "hold.sh: spare ${HOLD} cannot repeat hold ${REPEATS}: ${why}" >&2; exit 2; } fi if [ "${HOLD}" != smoke ] && [ -f "${RES}/node-image" ] && [ "$(cat "${RES}/node-image")" != "${H59_IMAGE}" ]; then echo "hold.sh: ${ROW}'s holds run node image $(cat "${RES}/node-image"), not ${H59_IMAGE}" >&2 @@ -89,6 +93,8 @@ if [ "${MEMMIN}" -gt 0 ] && [ "$(memavail_gib)" -lt "${MEMMIN}" ]; then log "MemAvailable $(memavail_gib) GiB (< ${MEMMIN}): no hold"; echo "memavail $(memavail_gib) GiB" > "${R}/no-hold"; exit 5 fi [ "${HOLD}" = smoke ] || [ -f "${RES}/node-image" ] || echo "${H59_IMAGE}" > "${RES}/node-image" +# From here the hold counts, also if it is interrupted before it records its rc. +date -u +%FT%TZ > "${R}/started" # A spare names the hold it stands in for only once it runs (h59.py reads `repeats`). [ -z "${REPEATS}" ] || echo "${REPEATS}" > "${R}/repeats" left=$(mine) diff --git a/bench/h59/run.py b/bench/h59/run.py index b6adc1e3..b0447d69 100644 --- a/bench/h59/run.py +++ b/bench/h59/run.py @@ -7,7 +7,11 @@ run.py hold HOLD [REPEATS] "MODE ARM ARM" of a registered hold: 1-3 perf, 4 quality, the spares 5 and 6 repeating hold REPEATS in its order, smoke as hold 1 - run.py spare-ok RESULTS K exit 0 when hold K ran and one of its arms was not measured + run.py spare-ok RESULTS K [SPARE] exit 0 when hold K ran and the record standing for it (hold K, + or spare 5 once it repeated K) has an arm not measured; spare 6 + only after spare 5 + run.py cpu HOST CPUINFO exit 0 when the CPU class (the model number the optimizer + matches a ledger row by) is the profile's, exactly run.py models G exit 0 when prep.sh left every file the arms read in G/models run.py render ARM OUTDIR `k3d-gpu agent manifest --model` for the arm in the hold's cluster (REPO, CLUSTER, KUBECONFIG, H59_PROFILE, G from the @@ -137,16 +141,47 @@ def hold(n, repeats=None): "(1-3 perf, 4 quality, a spare 5 or 6 names the hold it repeats)") -def spare_ok(results, k): - """A spare repeats hold k only when hold k ran (its rc is recorded) and one of its arms was - not measured.""" +def ran(d): + """A hold ran once it started (hold.sh writes `started` when the floors pass, before anything + runs) or recorded its rc.""" + return any(os.path.exists(os.path.join(d, f)) for f in ("started", "rc")) + + +def spare_ok(results, k, spare=None): + """A spare repeats hold k only when hold k ran and the record that stands for hold k now + (hold k, or spare 5 once it repeated hold k: h59.hold_dirs) has an arm not measured. Spare 6 + runs only after spare 5: h59.py reads spare 6 over spare 5 for the same hold, so the later + spare is the one it scores.""" + mode, arms = hold(str(k)) d = os.path.join(results, f"hold{k}") - if not os.path.exists(os.path.join(d, "rc")): + if not ran(d): return False, f"hold {k} has not run" - mode, arms = hold(str(k)) - if all(h59.status(os.path.join(d, a)) == "measured" for a in arms): - return False, f"every arm of hold {k} was measured" - return True, f"hold {k} has an arm not measured" + if str(spare) == str(SPARES[1]) and not ran(os.path.join(results, f"hold{SPARES[0]}")): + return False, f"spare {SPARES[1]} runs only after spare {SPARES[0]}" + cur = h59.hold_dirs(results)[int(k)] + where = f"hold {k}" if cur == d else f"spare {os.path.basename(cur)[4:]}, which repeated hold {k}," + if all(h59.status(os.path.join(cur, a)) == "measured" for a in arms): + return False, f"every arm of {where} was measured" + return True, f"{where} has an arm not measured" + + +def _model_opt(): + spec = importlib.util.spec_from_file_location("model_opt", os.path.join(REPO, "console", "model_opt.py")) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def cpu_class(cpuinfo): + """The CPU class the optimizer matches a ledger row by: console/model_opt.py class_tokens of + the first `model name` line, as `k3d-gpu` reads it ("6741P", "i9-13900T", "i7-13700").""" + name = "" + with open(cpuinfo) as f: + for line in f: + if line.startswith("model name"): + name = line.partition(":")[2].strip() + break + return " ".join(_model_opt().class_tokens(f"cpu:{name}")) if name else "" def models_ready(g): @@ -526,9 +561,13 @@ def main(argv): mode, arms = hold(a[0], a[1] if len(a) > 1 else None) print(mode, *arms) elif cmd == "spare-ok": - ok, why = spare_ok(a[0], a[1]) + ok, why = spare_ok(a[0], a[1], a[2] if len(a) > 2 else None) print(why) return 0 if ok else 1 + elif cmd == "cpu": + got = cpu_class(a[1]) + print(got) + return 0 if got.lower() == profile(a[0])["cpu"].lower() else 1 elif cmd == "models": missing = models_ready(os.path.expanduser(a[0])) for m in missing: diff --git a/bench/h59/test_run.py b/bench/h59/test_run.py index cc225cd8..f4a34108 100644 --- a/bench/h59/test_run.py +++ b/bench/h59/test_run.py @@ -297,7 +297,7 @@ def test_docker_argv_runs_the_jobs_container(self): self.assertIn("/g/models:/models", a) self.assertEqual(a[-8:], ["/scripts/convert.sh"] + spec["args"]) with self.assertRaises(SystemExit): - run.docker_argv(dict(spec, image="ghcr.io/ggml-org/llama.cpp:full-b11749"), "/m", "/s") + run.docker_argv(dict(spec, image=run.LLAMA_FULL_IMAGE.split("@")[0].replace("b11960", "b11749")), "/m", "/s") class Render(unittest.TestCase): @@ -407,7 +407,7 @@ def c(t): return t["spec"]["containers"][0] self.assertEqual(run.check("O", "desktop", base["O"]), []) self.assertEqual(run.check("L", "desktop", base["L"]), []) - self.assertTrue(bad("L", lambda t: c(t).update(image="ghcr.io/ggml-org/llama.cpp:server-b11749"))) + self.assertTrue(bad("L", lambda t: c(t).update(image=run.LLAMA_CPU_IMAGE.split("@")[0].replace("b11960", "b11749")))) self.assertTrue(bad("L", lambda t: c(t)["args"].remove("--spec-type"))) self.assertTrue(bad("L", lambda t: c(t)["args"].extend(["--model-url", "https://x"]))) self.assertTrue(bad("L", lambda t: c(t)["args"].__setitem__(c(t)["args"].index("--threads") + 1, "16"))) @@ -568,6 +568,7 @@ def test_holds_feed_the_scorer(self): ["idle", "n1", "n4", "ttft", "prefix"]) q = jl(os.path.join(res, "hold4", "O", "quality.json")) self.assertEqual((q["n"], q["errors"]), (50, 0)) + self.registered_requests(res, srv.bodies) rep = h59.score(res, "h57d") self.assertEqual(rep["verdict"], "SUPPORTED", json.dumps({k: v for k, v in rep.items() if k != "arms"})) self.assertEqual([rep["arms"][a]["n4_requests"] for a in ("L", "O")], [48, 48]) @@ -583,6 +584,37 @@ def test_holds_feed_the_scorer(self): srv.server_close() box.close() + def registered_requests(self, res, bodies): + """The requests as registered (H59 pre-registration, Measurement 1-3), per arm: in each + performance hold 3 warmup, 8 N=1 and 16 N=4 requests streamed at max_tokens 256 with + ignore_eos and temperature 0, 4 TTFT requests on H54's prompt at max_tokens 16, 6 prefix + agents at max_tokens 8; GSM8K items 0-49 at max_tokens 1024, temperature 0, thinking off. + Every load prompt is unique; each arm's boot limit is 900 s.""" + ttft = rd(os.path.join(ROOT, "bench", "h54", "ttft-prompt.txt")).rstrip("\n") + for k in (1, 2, 3): + for arm in ("L", "O"): + ad = os.path.join(res, f"hold{k}", arm) + for ph, n, c in (("warmup", 3, 1), ("n1", 8, 1), ("n4", 16, 4), ("ttft", 4, 1)): + rec = jl(os.path.join(ad, f"load-{ph}.json"))["phases"][0] + self.assertEqual((rec["concurrency"], len(rec["samples"])), (c, n), f"hold {k} {arm} {ph}") + self.assertEqual(jl(os.path.join(ad, "boot.json"))["limit_s"], 900) + for api, arm in (("v1", "L"), ("v3", "O")): + b = [x for p, x in bodies if p.split("/")[1] == api] + load = [x for x in b if x.get("stream") and "tools" not in x] + gen = [x for x in load if x["max_tokens"] == 256] + tt = [x for x in load if x["max_tokens"] == 16] + self.assertEqual((len(gen), len(tt), len(load)), (3 * 27, 3 * 4, 3 * 31), arm) + self.assertTrue(all(x.get("ignore_eos") is True and x["temperature"] == 0 for x in gen), arm) + self.assertTrue(all(x["temperature"] == 0 and x["messages"][0]["content"].endswith(ttft) for x in tt), arm) + prompts = [x["messages"][0]["content"] for x in load] + self.assertEqual(len(set(prompts)), len(prompts), arm) + pre = [x for x in b if "tools" in x] + self.assertEqual((len(pre), {x["max_tokens"] for x in pre}), (3 * 6, {8}), arm) + gsm = [x for x in b if not x.get("stream") and x.get("max_tokens") == 1024] + self.assertEqual(len(gsm), 50, arm) + self.assertTrue(all(x["temperature"] == 0 and x["chat_template_kwargs"] == {"enable_thinking": False} + for x in gsm), arm) + def test_crescent_n16_and_the_void_rerun(self): box = Box("crescent", cluster="k3dgpu-g-h59-1") srv = serve({"v1": 0.0004, "v3": 0.0004}) @@ -672,12 +704,25 @@ def test_wrong_cpu_class(self): self.assertEqual(r.returncode, 2) self.assertIn("not the i7-13700", r.stderr) + def test_cpu_class_is_the_exact_model_number(self): + # The optimizer matches a row's CPU class as an exact token (console/model_opt.py + # class_tokens), so a sibling SKU whose name only contains the row's number is another class. + for name, host, ok in (("13th Gen Intel(R) Core(TM) i7-13700K", "wildct", False), + ("13th Gen Intel(R) Core(TM) i7-13700", "wildct", True), + ("13th Gen Intel(R) Core(TM) i9-13900TE", "desktop", False), + ("13th Gen Intel(R) Core(TM) i9-13900T", "desktop", True), + ("Intel(R) Xeon(R) 6741P", "crescent", True), + ("Intel(R) Xeon(R) 6741P-B", "crescent", False)): + put(os.path.join(self.d, "cpuinfo"), f"processor\t: 0\nmodel name\t: {name}\n") + r = self.hold(host, "1") + self.assertEqual(r.returncode, 4 if ok else 2, (name, r.stderr)) + def test_unregistered_holds(self): for args in (("desktop", "7"), ("desktop", "5"), ("desktop", "01"), ("desktop", "1", "2"), ("nowhere", "1")): self.assertEqual(self.hold(*args).returncode, 2, args) def test_image_pinned_by_digest(self): - self.assertEqual(self.hold("desktop", "1", H59_IMAGE="cryptoandcoffee/k3d-gpu:0.1.80").returncode, 2) + self.assertEqual(self.hold("desktop", "1", H59_IMAGE=NODE_IMAGE.split("@")[0]).returncode, 2) def test_shim_and_models(self): os.remove(os.path.join(self.g, "bin", "k3d")) @@ -702,6 +747,32 @@ def test_spares_repeat_a_failed_hold_only(self): self.assertEqual(self.hold("desktop", "5", "3").returncode, 4) # passes to the lock self.assertFalse(os.path.exists(os.path.join(self.res, "hold5", "repeats"))) + def test_spare_six_follows_spare_five_and_reads_its_record(self): + # h59.py reads spare 6 over spare 5 for the same hold, so spare 6 runs only after spare 5, + # and a spare repeats a hold only while the record standing for it has an arm not measured. + self.ran(1, {"O": "measured", "L": "failed"}) + self.assertEqual(self.hold("desktop", "6", "1").returncode, 2) # spare 5 has not run + self.ran(5, {"O": "measured", "L": "measured"}) + put(os.path.join(self.res, "hold5", "repeats"), "1\n") + r = self.hold("desktop", "6", "1") + self.assertEqual(r.returncode, 2, r.stderr) # spare 5 measured hold 1 + self.assertIn("spare 5", r.stderr) + self.ran(5, {"O": "measured", "L": "failed"}) + self.assertEqual(self.hold("desktop", "6", "1").returncode, 4) # spare 5 failed too + self.ran(2, {"O": "not_deliverable", "L": "measured"}) + self.assertEqual(self.hold("desktop", "6", "2").returncode, 4) # spare 5 repeated hold 1 + + def test_an_interrupted_hold_counts(self): + # A hold killed after it started records no rc; it still ran, so it is not run again and a + # spare may repeat it. + hd = os.path.join(self.res, "hold3") + put(os.path.join(hd, "started"), "2026-10-29T00:00:00Z\n") + put(os.path.join(hd, "O", "status.json"), json.dumps({"status": "measured"})) + r = self.hold("desktop", "3") + self.assertEqual(r.returncode, 2) + self.assertIn("interrupted or not", r.stderr) + self.assertEqual(self.hold("desktop", "5", "3").returncode, 4) + def test_node_image_is_the_rows(self): put(os.path.join(self.res, "node-image"), "cryptoandcoffee/k3d-gpu:0.1.79@sha256:" + "b" * 64 + "\n") r = self.hold("desktop", "1")