From 78342bd6452f90328387a7620af24325c9185305 Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Thu, 29 Oct 2026 18:02:22 -0700 Subject: [PATCH 1/2] ledger(H62): pre-register k3d-gpu against Intel's EAS default, with its runner H62 asks H50's question against Intel's new default for the agent toolkit: EAS v1.2.0-rc.1 (5c27fb3) installing EAT 2.0 (release-2.0.0 036b662) and enterprise-inference release-1.1 (170eb3a) with Kubespray onto one node, serving its gemma-4-26b catalog entry. A1 is H50's A1 unchanged. Both arms run in KVM guests of one shape on crescent, because Kubespray on the host would remove Docker and the serving pairs, and EAS's configure refuses Ubuntu 26.04. The design, statistic, gates, scorer and verdict rule are H50's (rule L3, seed H62-crescent-2026-10-29). The deviations from stock are named: D1 pins the layer manifest's branches to the commits its own comments name, D2 serves the QAT checkpoint, D3 is the guest. A re-run fires when EAS v1.2.x or an EAT 2.x release is tagged. The runner is bench/h62: the guests' QEMU in a pinned container, each guest's setup, the setup and wiring-smoke gates, per-arm control for live.sh, the hold wrapper, the campaign and the teardown. live.sh gains a remote-arm branch for ARM=H62; the local arms run unchanged. pairs.py reads a pod cgroup through an exec: command, so the idle-arm check reads the pod inside its guest. serving.py records the guests' CPU as vm_cpu. The technical report counts 82 rows, 20 of them PENDING. --- .github/workflows/ci.yml | 1 + EXPERIMENTS.md | 332 +++++++++++ bench/eatflow/crescent/serving.py | 23 +- bench/eatflow/crescent/test_crescent.py | 28 + bench/eatflow/live.sh | 50 ++ bench/eatflow/pairs.py | 47 +- bench/eatflow/test_pairs.py | 78 +++ bench/h62/arm-A1 | 3 + bench/h62/arm-B5 | 3 + bench/h62/arm.sh | 89 +++ bench/h62/campaign.sh | 60 ++ bench/h62/guest-a.sh | 77 +++ bench/h62/guest-b.sh | 69 +++ bench/h62/h62.py | 528 +++++++++++++++++ bench/h62/pairs-hold.sh | 27 + bench/h62/qemu.Dockerfile | 12 + bench/h62/setup.sh | 103 ++++ bench/h62/smoke.sh | 46 ++ bench/h62/teardown.sh | 35 ++ bench/h62/test_h62.py | 738 ++++++++++++++++++++++++ bench/h62/upstream.sh | 11 + bench/h62/vm.sh | 104 ++++ docs/intel/technical-report.md | 4 +- 23 files changed, 2458 insertions(+), 10 deletions(-) create mode 100755 bench/h62/arm-A1 create mode 100755 bench/h62/arm-B5 create mode 100755 bench/h62/arm.sh create mode 100755 bench/h62/campaign.sh create mode 100755 bench/h62/guest-a.sh create mode 100755 bench/h62/guest-b.sh create mode 100644 bench/h62/h62.py create mode 100755 bench/h62/pairs-hold.sh create mode 100644 bench/h62/qemu.Dockerfile create mode 100755 bench/h62/setup.sh create mode 100755 bench/h62/smoke.sh create mode 100755 bench/h62/teardown.sh create mode 100644 bench/h62/test_h62.py create mode 100755 bench/h62/upstream.sh create mode 100755 bench/h62/vm.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 90f4bde3..1491561e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -268,6 +268,7 @@ jobs: - run: python3 bench/h54gpu/test_h54gpu.py - run: python3 bench/h57d/test_h57d.py - run: python3 bench/h59/test_h59.py + - run: python3 bench/h62/test_h62.py - run: python3 bench/admission/test_ctl.py - run: python3 bench/admission/test_analyze.py - run: bash bench/admission/test_run.sh diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index c9689b5f..760d7873 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -68,6 +68,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H50 | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) On EAT's own target CPU (Xeon 6741P, Granite Rapids, AMX), inside EAT's banking flow, the k3d-gpu CPU stack A1 (OVMS int4 ~~with the CPU Fast Draft~~, `agent manifest`; draft OFF per [amendment A4](#crescent-amendment-a4-a1-is-k3d-gpus-best-cpu-config-for-the-xeon-2026-10-20): H34x REFUTED it on this Xeon) still delivers far more B24 tasks within the 120 s turn budget than EAT stock B1 (vLLM CPU bf16, `xeon-values.yaml` as shipped, AMX on), both arms in one hold, interleaved. H49b's comparison moved to the hardware EAT is built for. A stack-vs-stack claim (amendment 16). | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement on the Xeon. A1 within-budget success >= 0.70 (0.8). B1 >= 0.60 (0.7): with AMX bf16 its turns should fit 120 s. Paired A1 - B1 in [-0.15, +0.20] (0.6). L2 matches >= 16 pairs and >= 75% (0.6), and the gates are read first, so the expected verdict is REFUTED (0.5), INCONCLUSIVE (0.45, mostly the L2 gates), SUPPORTED (0.05). Not gated: median attempt A1/B1 <= 0.7 (0.6). | H49b's design, statistic, outcome classes, grader, gates and thresholds on `crescent`; see [H50-H52 pre-registration](#h50-h52-pre-registration-the-xeon-6741p-2026-10-20). Holds of at most 40 min until 24 pairs or 8 holds. SUPPORTED: paired difference CI lower bound >= +30 points and A1 >= 0.6, with L2, the void and idle-arm checks passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. `bench/eatflow/crescent/pairs-hold.sh H50 ` (`LIVE_MODE=pairs LIVE_CAP=2370 live.sh H50`, KV 40 GiB), `agentbench pairs-report --hyp H50 --arm-a A1 --arm-b B1`. Model, arms, sampling and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20) (Gemma 4 26B-A4B QAT; breaks direct replication of H49b). Analysis rule L3 (primary) from [amendment A3](#crescent-amendment-a3-analysis-rule-l3-for-the-crescent-pairs-rows-2026-10-20). B1 is EAT release-1.1.0's stock (vLLM CPU v0.28.0) and runs only after a stall check, per [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); H50's deploy commit, wiring smoke and pre-hold records per [amendment E3](#crescent-amendment-e3-h50s-deploy-commit-wiring-smoke-and-pre-hold-records-2026-10-29). Result ([H50 run](#h50-run-xeon-6741p-2026-10-29)): E1's stall check passed (60 of 60 requests, no stall); 24 complete pairs in 1 of the 8 permitted holds, no void pair, idle check 0.0022 CPU, balance gate passed (smallest p 0.97). Within-budget success A1 19/24 (0.79), B1 21/24 (0.875); paired difference A1 - B1 -0.083 (95% CI -0.208 to 0.000), under +30 points, so REFUTED. Not gated: median attempt A1 8.0 s, B1 12.0 s; paired J per attempt A1/B1 0.81. | REFUTED | | H50o | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) H50's comparison with arm B3, EAT's own OVMS chart (`core/helm-charts/ovms` at release-1.0.3-hotfix, stock values, `openvino/model_server` 2025.4, the closest OpenVINO/ model it pulls, `OpenVINO/Qwen3-4B-int4-ov`) instead of B1: A1 delivers far more B24 tasks within budget than B3. With H50 it splits B1 -> A1 into EAT switching its own engine (B1 -> B3) and what k3d-gpu adds on top (B3 -> A1). | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement. B3's stock graph has no tool parser (the chart's `ovms --pull` writes none) and its model reasons by default, so its tool calls arrive as text: B3 within-budget success <= 0.10 (0.7), paired A1 - B3 >= +60 points (0.65). Pair yield: a B3 attempt should end within 30-90 s (a failed tool call or a reasoned text answer), so a 40-min hold should run 15-25 pairs and 24 pairs fit 2-3 holds; L2 passes (0.6), so SUPPORTED (0.5), INCONCLUSIVE (0.45). | As H50 with arm B = B3 (same pairs, gates, thresholds, statistic, cluster per hold `k3dgpu-e-h50o`); holds of at most 40 min until 24 pairs or 6 holds. The B1 -> B3 and B3 -> A1 splits are described from the two rows' matched pairs, never gated. `bench/eatflow/crescent/pairs-hold.sh H50o `, `agentbench pairs-report --hyp H50o --arm-a A1 --arm-b B3`. Model and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20). Analysis rule L3 from [amendment A3](#crescent-amendment-a3-analysis-rule-l3-for-the-crescent-pairs-rows-2026-10-20). B3 is unchanged under [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29) (EAT release-1.1.0's OVMS chart renders the same objects); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); H50o's deploy commit, B3's records and its wiring smoke per [H50o amendment O1](#h50o-amendment-o1-deploy-commit-b3s-records-and-a-b3-wiring-smoke-2026-10-29). Result ([H50o run](#h50o-run-xeon-6741p-2026-10-29)): B3 was not ready in hold 1, so under M1's rule H50o runs no further hold and no pair ran (1 of the 6 permitted holds). On k3d, B3's PVC bound, its pod was scheduled and its init container pulled `OpenVINO/gemma-4-26b-a4b-it-int4-ov` at `4e5885c3` (every file equal to the Hub's), but OVMS 2025.4's server failed on each of its 7 starts, going into CrashLoopBackOff; the last start's recorded exit is 139 (SIGSEGV), a second into loading the model. The same server in a plain container, outside k3d, on the same revision, exited 139 the same way with and without the chart's limits. Not deliverable: EAT's OVMS chart as shipped does not serve this model. | INCONCLUSIVE | | H51 | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) Graded agents per box on the Xeon 6741P: H45's ramp, gates and scoring (B15-RO, typical load, 300 s steps, pooled p95 turn <= 10 s, success >= the arm's N=1 success - 10 points, errors <= 1%, >= 60 window requests), extended to N=16, for A1 and B1. | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement. A1 in [1, 8] (0.5). B1 in [0, 4] (0.5). B1 <= A1 (0.55). | H45's `apb` runner and scorer (`apb-score --hyp H51`, ramp 1, 2, 4, 8, 16), one fresh cluster per hold, at most 4 holds of at most 40 min. A prediction holds when the arm's bracket lies inside its range and is refuted when it lies outside; B1 <= A1 is decided only on matched load. REFUTED when any prediction is refuted, SUPPORTED when all three hold, else INCONCLUSIVE. Runs only if H50's holds end within their cap. Model and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20). B1 as H50, from [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); deploy commit, M1's ranges in the scorer, one hold per arm and the wiring smokes per [H51 amendment S1](#h51-amendment-s1-deploy-commit-scorer-ranges-ramp-per-hold-and-wiring-smokes-2026-10-29). Result ([H51 run](#h51-run-xeon-6741p-2026-10-29)): one hold per arm, 2 of the 4 permitted, none VOID. Both arms failed N=1. A1: 65 window requests, turn p95 10.41 s against the 10 s SLO, success 30/32. B1: 58 window requests (under 60), p95 11.52 s, success 27/29. Agents per box A1 0, outside its predicted [1, 4], so REFUTED. B1's 0 lies inside [0, 4], and B1 <= A1 holds on matched load (0 <= 0). | REFUTED | +| H62 | Against Intel's new default. On the Xeon 6741P, inside EAT's banking flow (B24) with Gemma 4 26B-A4B on its QAT weights, the k3d-gpu CPU stack A1 (H50's A1 unchanged: `agent manifest`, OVMS 2026.4.1 int4 on the snapped QAT IR, CPU, no draft) delivers far more tasks within the 120 s turn budget than B5, Intel's documented single-node default for its agent toolkit: EAS v1.2.0-rc.1 (`5c27fb3`) installing EAT 2.0 (`release-2.0.0` `036b662`) and enterprise-inference `release-1.1` (`170eb3a`) with Kubespray onto one node, serving its catalog entry `gemma-4-26b` (vLLM CPU v0.28.0 on KServe, NRI balloons) on the same QAT weights. Each stack runs in its own KVM guest of one shape on the same host; both are live in one hold, interleaved. Deviations from stock are named (D1-D3). A stack-vs-stack claim (amendment 16). | Written 2026-10-29, before any H62 step and after H50 (A1 19/24, B1 21/24) and H51 were known. A1 within-budget success >= 0.70 (0.65). B5 >= 0.70 (0.6). Paired A1 - B5 in [-0.20, +0.15] (0.6). Balance gate passes (0.9). REFUTED (0.6), INCONCLUSIVE (0.35: mostly B5 not deliverable in its guest within 120 min, a failed wiring smoke or a gate), SUPPORTED (0.05). Not gated: median attempt A1/B5 <= 0.85 (0.5); A1's median attempt in its guest over H50's 8.0 s on the host in [1.0, 1.3] (0.6). | [H62 pre-registration](#h62-pre-registration-k3d-gpu-against-intels-new-default-eas-with-eat-20-on-the-xeon-6741p-2026-10-29): H50's design, statistic, gates and verdict rule (rule L3, amendment A3, seed `H62-crescent-2026-10-29`), two KVM guests on crescent (VM a 96 vCPU and 96 GiB for A1, VM b 96 vCPU and 192 GiB for B5), the harness's LiteLLM per arm on the host, holds of at most 40 min until 24 pairs or 8 holds, each hold restarting both serving pods. Setup and wiring-smoke gates (memory, disk, AMX in each guest, NFD label, B5's weights equal to the QAT checkpoint, EAS's own `validate` and gateway check) come before hold 1. SUPPORTED: paired difference CI lower bound >= +30 points and A1 >= 0.6, with every gate passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. `bench/h62/campaign.sh`; `agentbench pairs-report --hyp H62 --arm-a A1 --arm-b B5`. Re-run as a new row when EAS v1.2.x or an EAT 2.x release is tagged (`bench/h62/upstream.sh`). | PENDING | | H52 | A plain `k3d-gpu up` (no flags, the auto-adapt default) on the Xeon 6741P host as found detects and enables every engine actually present and advertises each usable one to the scheduler, with nothing missing and nothing phantom; engines present but not configured on the host are named, not advertised. | Written 2026-10-20 from an inventory read from sysfs, lspci and CPUID, not from k3d-gpu. 4 x Arc Pro B70 (xe) as `gpu.intel.com/xe: 4`; AMX TILE, BF16, INT8, FP16 and 14 AVX-512/VNNI labels; no hybrid, NUMA, NVIDIA, Gaudi or NPU objects; DSA, IAA and QAT named as present but unconfigured (0.4: `up` may not name them). | The table in [H50-H52 pre-registration](#h50-h52-pre-registration-the-xeon-6741p-2026-10-20): `k3d-gpu doctor`, `K3D_GPU_CLUSTER=k3dgpu-h52 k3d-gpu up`, `kubectl get node -o json`, `k3d-gpu test`, and one enumeration pod per B70. SUPPORTED: every row matches. REFUTED: any row does not. Run 2026-10-20 ([H52 run](#h52-run-xeon-6741p-2026-10-20)): the four B70s (`gpu.intel.com/xe: 4`, `xmx=true`), AMX (TILE, BF16, INT8, FP16) and all 14 AVX-512/VNNI labels detected and reached pods, nothing phantom; but no media engine label on `xe`, and `up` does not name the unconfigured DSA, IAA and QAT (`doctor` does). | REFUTED | | H52b | After `k3d-gpu host-setup --apply` configures the host's DSA, IAA and QAT (the fix that adds that path, merged first), a plain `k3d-gpu up` also advertises them, and each one reaches a pod; the host is then restored exactly. | Written 2026-10-20, before the fix exists. DSA, IAA and QAT allocatable > 0, each at the count host-setup reports configuring; their test pods see `/dev/dsa`, `/dev/iax`, `/dev/vfio` (0.6). The other rows read as H52's expected table, media labels included ([amendment D1](#h52-amendment-d1-the-fixes-and-h52b-2026-10-20)). | Runs after H52, once the host-setup fix is merged; same procedure plus `host-setup --apply` before `up` and its printed revert after. SUPPORTED: the three engines advertised and reached, the H52 rows unchanged, and the host inventory read back equal to the one taken before. REFUTED: any of these fails. The host-setup command (`--apply --with-kernel-wqs`), the plan read first, the counts, the plain `up`'s node image and the revert per [amendment D2](#h52-amendment-d2-h52bs-host-setup-step-on-crescent-2026-10-29); the runner and deploy commit per [amendment D3](#h52-amendment-d3-h52bs-runner-and-deploy-commit-2026-10-29). Result ([H52b run](#h52b-run-xeon-6741p-2026-10-29)): one hold at k3d-gpu 0.1.79; the plan passed D2's gate (8 commands) and applied. DSA advertised 3 of the 3 configured work queues and its pod saw `/dev/dsa`; IAA got no work queue and no resource (as D2 item 4 disclosed); QAT got 48 VFs, but its plugin crash-looped and advertised none; `gpu.intel.com/xe` read 2, not 4: the serving pairs hold about half the VRAM of all four cards, and `up`'s busy-card filter (half the VRAM) left out the two just over half (16.1, 16.2 GiB) and kept the two just under it (15.4, 15.5 GiB), which the pairs also hold, so two enumeration pods ran, not four. Media labels (`vcs` 2, `vecs` 2), AMX, the 14 AVX-512/VNNI labels, CPU 96 and no other vendor read as expected. The third inventory equals the first. Three clauses fail, so REFUTED. | REFUTED | | H53 | Inside EAT's banking flow on the Arc Pro B70s with Qwen3.8-27B (thinking off), k3d-gpu's own GPU choice AG (OVMS int4 on one card, `agent manifest`, auto device) delivers far more B24 tasks within the 120 s turn budget than B2, "EAT PR #21 nearest profile (Qwen3.6-27B) on Qwen3.8 weights, fp16" (vLLM XPU v0.28.0, TP 4). A stack-vs-stack claim. | Written 2026-10-20, before any H53 measurement. AG >= 0.70 (0.6); B2 boots by ladder rung (b) (0.6); B2 >= 0.50 (0.5); block mean AG - B2 in [-0.15, +0.30] (0.5). INCONCLUSIVE (0.6), REFUTED (0.3), SUPPORTED (0.1). | Solo holds in the order AG, B2, B2, AG, AG, B2 (up to 8), blocks = hold pairs (2k-1, 2k), block-level inference: mean of per-block d over >= 3 counted blocks, t-interval. SUPPORTED: lower bound >= +30 points and AG >= 0.6, L3 gates passed. REFUTED: upper bound < +30 points. Else INCONCLUSIVE. `bench/eatflow/crescent/solo-hold.sh N ARM`, `agentbench solo-report --hyp H53 --arm-a AG --arm-b B2`. See [H53 pre-registration](#h53-pre-registration-eats-banking-flow-on-arc-pro-b70-2026-10-20). | PENDING | @@ -4199,6 +4200,337 @@ are byte-identical across the two versions. **Disclosed.** Written with the predictions of the covered rows in view and before any of their data existed. +### H62 pre-registration: k3d-gpu against Intel's new default, EAS with EAT 2.0, on the Xeon 6741P (2026-10-29) + +Written 2026-10-29 and committed with its runner (`bench/h62/`) before any H62 step. No guest +exists on crescent, nothing has been installed there for H62, and nothing below is a measurement. +Crescent was only read, over read-only ssh, at 2026-10-30T00:04Z. The predictions in the row are +never edited. A protocol change is a dated amendment, merged before the data it governs. + +**Why.** + +- The box owner, 2026-10-29: EAS becomes Intel's default once full support lands, so k3d-gpu + should be benchmarked against it too. +- H50, H50o and H51 compared k3d-gpu's CPU stack with EAT release-1.1.0's stock arms, deployed by + EAT's own release-1.x charts. H50 is REFUTED (A1 19/24 within budget, B1 21/24) and so is H51. +- EAT 2.0 has no installer of its own. EAT `release-2.0.0`'s README ("Installing this release") + says the Agent Toolkit installs through the Intel AI for Enterprise Solutions (EAS) installer: + `./es_auto_installer.sh configure`, `init agent-toolkit`, `install agent-toolkit`. +- H62 asks H50's question against that path: Intel's documented single-node default, run as + documented, at pinned commits. + +**Heads, read 2026-10-29 (`gh api`, `git ls-remote`, the Hub API, the registries).** + +- EAS: main `97f6157d6a72f7173c698c474d4e51d2c79f1fd6`. `v1.2.0-rc.1` is a pre-release + (2026-10-27) at `5c27fb39a95eecec41aa4e29a1188984e9c726f3`, which is also the head of branch + `release-1.2`. Its `VERSION` file reads 1.1.0, a known issue its release notes name. No EAS + v1.2.0 tag exists. +- EAT: `release-2.0.0` (EAT #19 merged into it) is at `036b66234801e9bd950afd90830821859780d668`, + with no tag. EAT's newest tag is `release-1.1.0` (`0c726f57`). +- enterprise-inference (EI): `release-1.1` is at `170eb3abb1956de0787727f21e356aef2b13ff24`. The + rc.1 release notes' component table names exactly these three commits. +- Hugging Face: `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized` head is still + `f1e06dc520982d9b9edd76859fdb7ab209449949` (amendment M1's revision). `google/gemma-4-26B-A4B-it` + head is `4d7ae4984b7db7de8f8457170b3f1a419ee76d52`. Neither is gated. +- `vllm/vllm-openai-cpu:v0.28.0` serves index digest + `sha256:197354e6475e638eb5ccca09beb078488046a82de0e204e7429bf1a804df9647`, the image of + amendment E1's B1. + +**Intel's documented default for one Xeon box, read from source.** + +- The install path. EAT `release-2.0.0` `docs/single-node-deployment.md`, Step 1: + `git clone -b release-1.2` of EAS, `./es_auto_installer.sh configure`, `init agent-toolkit`, the + edits to `env/local/global_config.yaml` and `env/local/config.agent-toolkit.yaml`, then + `install agent-toolkit` ("Estimated time: 20-40 minutes on a fresh node", line 112, and "the vLLM + pod may take an additional 10-15 min to pull the model weights", line 127). The rc.1 release notes + say to `git clone -b v1.2.0-rc.1` and follow that guide from Step 1. +- Kubernetes is provisioned by the installer by default (lines 36-37). The README's "Deployment + Platform" section names "Kubernetes (Default)". EAS's shipped inventory is "Option A: Localhost + (default, single-node PoC)" with `ansible_connection: local` (`inventory/hosts.yaml` lines 8-22). + EAS v1.2.0-rc.1 installs Kubernetes 1.34.3 with Kubespray v2.30.0 and containerd + (`roles/kubernetes/defaults/main.yaml` lines 18, 25 and 94). +- EAT's Docker Compose path (`docker/deploy.sh`) is for "Development, demos, single machine" in + the README's table. It is not EAS and not Intel's default, so it is not an arm. +- The serving default (EI `release-1.1`, `model_manager/models.yaml`): + - `defaults.require_amx: true` (line 20), rendered as a required node affinity on + `feature.node.kubernetes.io/cpu-cpuid.AMXBF16=true` (`core/renderer.py` lines 90-95). + - A KServe `LLMInferenceService` (KServe v0.18.0, EI's `roles/kserve`), behind Envoy's AI + gateway with endpoint picking, registered in LiteLLM. + - Entry `gemma-4-26b` (lines 360-380): `google/gemma-4-26B-A4B-it`, `cpu: auto`, memory 128Gi, + vLLM CPU `server_version` 0.28.0, args `--block-size 128`, bf16, `mp`, chunked prefill, + `--enforce-eager`, prefix caching, tool parser gemma4, TP 1 and no `--max-model-len`; env + `VLLM_CPU_SGL_KERNEL` "0" and `VLLM_CPU_KVCACHE_SPACE` "40". + - The runtime's common env adds `VLLM_CPU_NUM_OF_RESERVED_CPU` "1" and `KMP_BLOCKTIME` "1" + (lines 557 and 560); the `llm` category adds `--max-num-seqs 128 --max-num-batched-tokens + 2048` (line 585). The image is `docker.io/vllm/vllm-openai-cpu:v0.28.0`, by tag (line 571). + - Under NRI balloons (EAS's default `kubernetes_cpu_policy`), `cpu: auto` takes the free + physical cores of one NUMA node (single-node-deployment.md, Step 1b). +- Gemma 4 26B-A4B is in the stock catalog (`gemma-4-26b`, row 26 of the guide's model table) but + is not the default model: EAT's `config.yaml` line 35 autodeploys `qwen2-5-coder-14b`. With the + guardrail on (the default with `auth_provider: litellm`, `config.yaml` line 26), Qwen3Guard-Gen-0.6B + deploys too (8 CPU, 16Gi, `models.yaml` lines 412-430). + +**Why both arms run in KVM guests.** + +- Kubespray on crescent's host would remove Docker. With `container_manager: containerd` and + docker.service running, Kubespray v2.30.0 runs its "Uninstall docker" block + (`roles/container-engine/validate-container-engine/tasks/main.yml`): Docker's reset kills every + container, runs `docker ps -aq | xargs -r docker rm -fv`, stops docker and containerd, removes + the packages and deletes `/var/lib/docker`. That would remove the two serving pairs, which the + box owner's grant says never to remove. +- EAS's `configure` refuses Ubuntu other than 22.04 and 24.04 (`es_auto_installer.sh` lines + 369-378). Crescent runs Ubuntu 26.04.1. +- A bring-your-own kubeconfig onto a k3d cluster is not stock either: unpatched EAS does not + deploy on k3s or k3d (k3d-gpu's overlay patches 0002 and 0006, PRs #248 and #260). That path + is k3d-gpu's own `eas up`, a different arm (A-EAS, below). +- So B5 runs Intel's default path unmodified, on a supported OS, in a disposable KVM guest on + crescent (Ubuntu 24.04), isolated from the host's Docker serving. +- A1 runs in a guest of the same shape. A1 on bare metal against B5 in a guest would hand + k3d-gpu an edge that comes from virtualization, not from either stack. A1 in its guest + against H50's A1 on the host is reported as a description of the VM tax, never gated. + +**Arms.** + +- **A1**: H50's A1 unchanged (amendments M1 and A4), in VM a. + - `k3d-gpu agent manifest` with `K3D_GPU_AGENT_DEVICE=CPU`, OVMS 2026.4.1, the snapped QAT IR + (`k3dgpu_gemma-4-26B-A4B-it-qat-int4-sym-g32-ov`, INT4 symmetric, group 32), tool parser + gemma4, no reasoning parser, no draft, no autoquant, as `live.sh`'s `gemma4` model set gives + it. + - The IR reaches the guest read-only (virtfs) and is checked against its `manifest.json` before + the copy into the node, as H50's was. + - `k3d-gpu up` runs in the guest at the deploy commit, with H50's node image + (`cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa...`) and amendment C1's kubelet eviction + thresholds. Docker CE 5:29.9.2 (containerd.io 2.4.2) comes from Docker's apt repository, + pinned, its key's fingerprint (`9DC85822...0EBFCD88`) checked first. k3d v5.9.0 and kubectl + v1.37.2 are the crescent workspace's checked binaries (amendment E4). + - VM a has no GPU, so k3d-gpu's auto device would pick the CPU too; the device stays named, as + in H50. Amendment A4's record runs once at setup: `k3d-gpu plan --device cpu --model + google/gemma-4-26B-A4B-it --explain` in VM a. A configuration other than A1's is recorded and + does not change A1. +- **B5**: Intel's documented single-node install in VM b, with deviations D1-D3 only. The steps, + in order: + 1. The node kernel settings of EAT's `docs/prerequisites.md` ("Node Kernel Settings"), verbatim: + `fs.inotify.max_user_instances = 8192` and `fs.inotify.max_user_watches = 1048576`. + 2. `git clone -b v1.2.0-rc.1`; the head must be `5c27fb3`. + 3. D1. + 4. `./es_auto_installer.sh configure --force`. `--force` is the installer's documented answer to + configure's one y/N prompt ("required in CI, where there is no terminal"); configure's own + steps are unchanged. + 5. `init agent-toolkit`. The two cloned layers must be at `170eb3a` and `036b662`. + 6. The one documented edit: `llm_services_autodeploy_models: ["gemma-4-26b"]` in + `env/local/config.agent-toolkit.yaml` (a value of the guide's model table). Then D2. + 7. `install agent-toolkit`, with `HF_TOKEN` unset (neither model is gated; recorded as set or + unset). + 8. Wait for the gemma-4-26b workload pod to be Ready. + + Left at EAS's and EAT's defaults: Kubespray onto localhost, NRI balloons, Istio ambient, + MetalLB, Envoy's AI gateway, observability, LiteLLM with the content-safety guardrail and the + hybrid router, Langfuse, Redis Stack, pgvector and agent-sandbox v0.5.0. + `base_domain_name` stays EAS's default (`solutions.ai`): the guide's example value is a + placeholder, and every request below resolves the gateway's name to its address directly. + +**Deviations from stock, named.** + +- **D1, commits.** The rc.1 layer manifest (`configs/repos/repos.agent-toolkit.yaml` lines 16 and + 25) names branches (`release-1.1`, `release-2.0.0`), with the commits in comments on the same + lines. D1 rewrites those two `rev:` values to the commits the comments name (`170eb3a`, + `036b662`), the release notes' component table, before `init`. Nothing else in the file + changes. `h62.py eas-pin` refuses a manifest whose branch or comment differs. +- **D2, weights.** `gemma-4-26b`'s `model_id` in `env/local/models.yaml` (the catalog `init` + copies, the file the guide points to for model settings) becomes + `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized`, so both arms serve the same QAT-trained + weights. This is the same deviation as amendment M1's B1. Every other field of the entry, and + every other entry, is unchanged. EI's download job fetches the weights with `hf download` and no + revision; the revision it fetched is read from the download's metadata and recorded, and the + weight gate checks the files. +- **D3, the guest.** Virtualization itself. + - Both guests share one shape: KVM, `-cpu host`, 96 vCPUs as 1 socket of 48 cores with 2 + threads (crescent's topology), unpinned vCPUs and no hugepages. VM a has 96 GiB, VM b 192 GiB + (B5's stack requests about 163 GiB: gemma 128, Qwen3Guard 16, router 2 and about 17 for the + rest, per the guide's formula, line 84). + - User-mode networking, with each guest's ssh and serve ports forwarded to 127.0.0.1 only. MetalLB's + single-node pool is the node IP /32 (EAS `roles/metallb/tasks/install.yaml` line 200): here + `10.0.2.15`, which does not overlap Calico's `10.233.64.0/18`. + - Ubuntu 24.04 cloud image `release-20261017`, sha256 `04e8f246...6c94`. cloud-init keeps the + image's package state: no upgrade at boot, and unattended-upgrades and the apt timers are off, + so no apt run competes with the installs. + - QEMU 10.2 (`qemu-system-x86` 1:10.2.1+ds-1ubuntu3.3) runs in a container built from + `ubuntu:26.04@sha256:6a06111c...`, with `--device /dev/kvm`, so the host gets no package. QEMU + passes AMX through from 7.0 on, and the host kernel (7.0) supports AMX for KVM guests. + +**The harness, shared by both arms, as in H50.** + +- On the host, per arm: the harness's LiteLLM (`v1.83.14-stable.patch.2@sha256:73cbb215...`), + the recording proxy (profile `agent_gemma4`), the banking agent and the MCP sidecar from + blueprint `758d484`. +- Each arm's server is reached through a `kubectl port-forward` of its Service inside its guest, + forwarded to the host. For A1 that is `svc/ovms`. For B5 it is + `gemma-4-26b-kserve-workload-svc:8000`, the Service EI's own LiteLLM bypass uses. +- B5's route therefore skips EAS's Envoy AI gateway, Istio, LiteLLM and the guardrail. H50 + compared engines behind one constant gateway, and with one serving replica the endpoint picker + has one endpoint. EAS's gateway and guardrail are exercised in the wiring smoke and are not + gated. + +**Model and sampling.** Amendment M1's, unchanged. Both arms serve the QAT-trained weights: A1 +the snapped INT4 IR, B5 the bf16 safetensors. Sampling is profile `agent_gemma4` (temperature +1.0, top_p 0.95, top_k 64, max_tokens 1024, thinking off), pinned by the recording proxy for +both arms. B5 serves without `--max-model-len` (the model's 262144), where H50's B1 had 8192; B24's +prompts fit both, and the difference is part of stock B5. + +**Workload, design, statistic and verdict: H50's.** + +- B24 at N=1 under `typical` background load (`bench/load/load.py` on the host's CPUs 24-47 and + 72-95), both arms live in one hold, interleaved pairs. +- Rule L3 (amendment A3) with seed `H62-crescent-2026-10-29`. Each pair's arm order is the seeded + coin, L3's successor to the A B B A order: a pure function of the seed and the pair index, so a + resumed hold draws the same order. +- Each attempt is preceded by the settle gap and the 10 s pre-window. The gates are the balance + gate, at least 16 complete pairs, at most 2 void pairs, and the idle-arm check. The idle-arm + check reads the other arm's serving pod inside its guest (`pairs.py`'s `exec:` cgroup read + through `bench/h62/arm.sh`); the median must stay at or under 0.5 CPU. +- 120 s turn budget, 240 s task cap, and H49b's outcome classes and grader. +- Scorer: `agentbench pairs-report --hyp H62 --arm-a A1 --arm-b B5`, H50's banking scorer + unchanged (`pairs.DESIGNS["H62"]` is `("H49", "L3")`). +- **SUPPORTED** when the paired difference's 95% CI lower bound is at least +30 points and A1's + within-budget success is at least 0.6, with every gate passed. **REFUTED** when the CI's upper + bound is under +30 points. Otherwise **INCONCLUSIVE**, including a row that is not deliverable. +- Reported, never gated: + - each arm's median attempt and turn p95, and the paired J per attempt (RAPL package energy, + which now includes both guests); + - the period-balanced estimate and L2's matched subset; + - the serving covariate, plus each guest's CPU-s/s (`serving.py`'s `vm_cpu`); + - B5's rendered CPU request, cpuset, `nproc` and the ISA its pod sees, per hold; + - the VM tax: A1 in its guest against H50's A1 on the host, within-budget success and median + attempt (`h62.py vmtax`). + +**Holds, caps and gates.** + +1. **Runner first.** This PR: the runner and its tests, before any guest exists. +2. **Deploy commit.** Every step runs from one clean, detached checkout at + `~/k3dgpu-bench/k3d-gpu-h62`, at an `origin/main` commit at or after this PR's merge, and + with #260 (whose `integrations/eat/eat-check.sh` has the EAS mode the smoke uses). If #260 has + not merged when crescent is ready, a run branch merges #260's head into main, as amendment E3 + did, and the run section names it. VM a gets the same commit (`git archive`). The commit goes + into every record. +3. **Setup phase**, not a hold, inside `flock -w 7200 ~/k3dgpu-bench.lock` (`bench/h62/setup.sh`). + - Host gate. MemAvailable must cover both guests plus 48 GiB (336 GiB; 357 GiB was available + at the feasibility read). `/` must be at most 90% used. Hugging Face, Docker Hub and + registry.k8s.io must answer. It is re-read every 10 min for up to 60 min. + - If the host gate still fails, the two qwen38 serving pairs are stopped under the box owner's + grant (`gpu-serving.sh stop`: `docker stop`, never `rm`, refused while aiperf runs) and setup + runs once more. After the teardown they are started again and checked through Dynamo + `:8100`, as amendment E2 item 7 does; that check gates nothing. + - A1's IR is checked against its manifest. The Hub's file list of the QAT checkpoint at + `f1e06dc5` is fetched for B5's weight gate. + - Guest gate: in each guest, `amx_tile` on 96 of 96 vCPUs and `nproc` 96. A failure means the + row is not deliverable (INCONCLUSIVE). + - Ceilings: VM a's setup (`guest-a.sh`) 45 min, VM b's install (`guest-b.sh`) 120 min, run side + by side. Intel's own estimate is 20-40 min plus 10-15 min of weights; Gate 2's inference-only + install took 863 s (`docs/intel/upstream-eat.md`, test plan row 2). An overrun means + INCONCLUSIVE ("not deliverable"), recorded as a finding. + - NFD gate: VM b's node carries `cpu-cpuid.AMXBF16=true`. + - Weight gate: every LFS file in B5's model directory equals the Hub's sha256 at `f1e06dc5`, + read in B5's pod. + - A failed NFD or weight gate means no hold runs until a dated amendment. +4. **Wiring smoke** (`bench/h62/smoke.sh`, E3-style). It is discarded, disclosed in the run + section, and not one of the 8 holds. + - In VM b, under the lock: `./es_auto_installer.sh validate agent-toolkit`, then + `integrations/eat/eat-check.sh` with `EAT_MODE=eas` through EAS's own gateway on + `gemma-4-26b`: a chat, a get_weather tool call, and a sandbox run that must print 385. + - Then one hold, `H62-smoke`, with the pair-start guard past its end: both serving pods + restart and answer, both warm-ups answer, no pair starts. + - All of it must pass before hold 1. A failure means no hold until a dated amendment. +5. **Holds** (`bench/h62/pairs-hold.sh N` under `bench/eatflow/crescent/hold.sh`). + - At most 40 min each (`LIVE_CAP=2370` inside `timeout -k 30 2400`), until 24 pairs or 8 holds. + - Each hold restarts both serving pods, then needs both Ready with at least 900 s of the hold + left. A hold that misses that is "not deliverable": it counts as one of the 8 and runs no + pair. After two such holds in a row, no further hold runs (INCONCLUSIVE, not deliverable). + - The 600 s pair-start guard applies. + - Hold-start rule (shared rule 1): with load1 over 12 or an aiperf process running, poll every + 2 min for up to 60 min, then run the hold flagged. + - The serving pairs are not stopped (unless the setup fallback above stopped them), and the + serving covariate is recorded. + - For scale: H50's one hold made 24 pairs between 07:16:33Z and 07:37:07Z, so 1 or 2 holds is + the likely count. +6. **Teardown** (`bench/h62/teardown.sh`), after the last hold or any abort. Both guests are + powered off and removed, and their disks, the cloud image and the QEMU image are deleted. The + results stay. +7. **One hold at a time** on crescent (amendment E2 item 4). The guests stay up from setup to + teardown, and other rows run no hold in that window. A hold of another row that does run then + sees the guests as foreign serving load: their containers carry no `k3dgpu` prefix, so its + `serving.py` counts them, and `hold.sh`'s cleanup never removes them. +8. **Records** that can hold a secret (the install logs, which handle the LiteLLM master key and the + sandbox router token) stay on crescent. Every record committed with the run section is first + scanned for those values (F3's secret scan). + +**Risks, named before measuring.** + +- AMX passthrough and the VM tax are unmeasured on crescent. The guest gate catches the first; + the VM tax is described. +- Kubespray rc.1 on one node behind user-mode networking has never run here, and that networking + carries roughly 60 GB of images and weights into VM b. +- VM a reads A1's IR over virtfs (9p). If the cloud image's kernel lacks the 9p modules, + `guest-a.sh` stops at its IR step, and an amendment names the fix before any hold. +- EI has no weights-revision pin, and EAS pulls the vLLM image by tag. The weight gate covers the + first; each hold records the pod's image ID for the second. +- Qwen3Guard deploys first, so `cpu: auto` sizes gemma's balloon to the cores left after it. The + deploy order and B5's rendered request and cpuset are recorded. +- Newer upstream releases not used, so stock pins stay: Kubespray v2.32.0 (stock v2.30.0) and vLLM + v0.32.0 (stock v0.28.0). +- `qemu-system-x86` 1:10.2.1+ds-1ubuntu3.3 is the archive's current `-updates` version. If it is + superseded before the image is built, the build fails and an amendment re-pins. +- Both guests' vCPUs float over all 96 host CPUs, the load CPUs included. VM b is never idle at + the OS level: Kubernetes, Istio, observability and Qwen3Guard run throughout. That background + applies to both arms' attempts alike and is inside every pre-window. + +**Re-run trigger.** + +- `bench/h62/upstream.sh` (`h62.py trigger`) reads both repositories' tags. It fires on an EAS + v1.2.x release tag (not a release candidate) or an EAT 2.x release tag (`release-2.X.Y` or + `v2.X.Y`). It is run before setup and before each hold, and its output goes into the records. +- Fired before hold 1: a dated amendment re-pins B5 to the tagged release, by amendment E1's logic + (EAT stock is what Intel ships now). If the tag's components equal the pinned commits, the + amendment only records that. +- Fired after hold 1: H62's holds finish at the pinned commits, and H62 keeps its verdict. A new + row, H62r, is pre-registered: the same design with B5 at the tagged release (its release notes' + component table). It runs after H62. + +**A-EAS, later, not this row.** k3d-gpu's own EAS path: +`k3d-gpu eas up --agent-toolkit --eas-ref v1.2.0-rc.1` (PR #260) with `gemma-4-26b`, on a k3d-gpu +cluster. Its `pins.json` profile `v1.2.0-rc.1` names the same EAS, EI and EAT commits as B5's D1, +so A-EAS and B5 run the same EAS. It uses a bastion and a bring-your-own kubeconfig, with no +Kubespray, so it can run on crescent's host. It is pre-registered as its own crescent F row after +#248-#251 and #260 merge and F3 passes on the desktop: F3's checks plus `validate agent-toolkit`, +with an optional descriptive B24 against B5. Its expected choices on crescent: `nri-balloons` +(patch 0003: no baked balloons on a single-NUMA, non-hybrid node), `require_amx: true` and +`nfd_enabled: false`. + +**Runner (this PR), with regression tests.** + +- `bench/eatflow/pairs.py`: `DESIGNS["H62"] = ("H49", "L3")` and its order seed. A pod cgroup + may be `exec:`, a command that prints a pod's `cpu.stat` on another machine, and + `pairs.py usage` reads each spec once. +- `bench/eatflow/live.sh`: `ARM=H62` in `LIVE_MODE=pairs` drives two remote arms + (`LIVE_REMOTE_A`, `LIVE_REMOTE_B`): restart, the 900 s readiness rule, records, the + port-forward and the cgroup spec. From the LiteLLM gateways on, the hold is H50's code. The + local arms run unchanged in the other branch. +- `bench/eatflow/crescent/serving.py`: `SERVING_VMS` records the guests' CPU as `vm_cpu`, outside + the serving covariate. +- `bench/h62/`: + - `h62.py`: the pins, the guests' shapes, QEMU command lines and cloud-init, the gates, D1, the + documented edit and D2, the pod picks, the weight-check script, the re-run trigger and the VM + tax. + - `vm.sh`, `qemu.Dockerfile`, `guest-a.sh`, `guest-b.sh`, `setup.sh`, `smoke.sh`, `arm.sh` (with + `arm-A1` and `arm-B5`), `pairs-hold.sh`, `campaign.sh`, `teardown.sh` and `upstream.sh`. +- Tests: `bench/h62/test_h62.py`, `H62Test` in `bench/eatflow/test_pairs.py` and `ServingVmTest` + in `bench/eatflow/crescent/test_crescent.py`. + +**Reproduce (planned).** On crescent, from `~/k3dgpu-bench/k3d-gpu-h62` at the deploy commit: +`bench/h62/upstream.sh`, then `bench/h62/campaign.sh` (setup, smoke, holds, score, teardown). +Records go to `~/k3dgpu-bench/results/H62/`, with the hold records in +`~/k3dgpu-bench/runs/H62-hold/` and the smoke in `~/k3dgpu-bench/smoke/H62/`. They come back +under `bench/eatflow/results/2026-10-29/H62/`. + ### H53 pre-registration: EAT's banking flow on Arc Pro B70 (2026-10-20) Written 2026-10-20, before any H53 measurement and before either H53 arm has run on a GPU. The diff --git a/bench/eatflow/crescent/serving.py b/bench/eatflow/crescent/serving.py index cd61ab74..b086dc67 100755 --- a/bench/eatflow/crescent/serving.py +++ b/bench/eatflow/crescent/serving.py @@ -7,7 +7,12 @@ Slot mode (H56, SERVING_SLOT=i with SERVING_CPUS = the slice): only slot i's containers are this hold's, so a sibling hold's cluster and load count as foreign; each row also carries own_busy_pct / other_busy_pct (/proc/stat over the slice and the rest of the -online CPUs) and own_mhz (mean scaling_cur_freq over the slice).""" +online CPUs) and own_mhz (mean scaling_cur_freq over the slice). + +VM mode (H62, SERVING_VMS = comma-separated container names): those containers run this +campaign's KVM guests, so each row records their CPU-s/s as vm_cpu and leaves them out of cpu +and serving_cpu. Their names carry no k3dgpu prefix, so another row's hold counts them as +foreign serving load.""" import json import os import signal @@ -40,6 +45,19 @@ def _slot(): SLOT = _slot() + + +def vms_from(value): + """SERVING_VMS: the container names of this campaign's guests (H62), or ().""" + return tuple(v.strip() for v in value.split(",") if v.strip()) + + +def split_vms(cpu, vms): + """(serving cpu, vm cpu): the readings of the containers named in vms move to the second.""" + return ({n: v for n, v in cpu.items() if n not in vms}, {n: v for n, v in cpu.items() if n in vms}) + + +VMS = vms_from(os.environ.get("SERVING_VMS", "")) stop = False @@ -113,8 +131,11 @@ def main(out): for n in cs if cur.get(n) is not None and prev.get(n) is not None} with open("/proc/loadavg") as la: load1 = float(la.read().split()[0]) + cpu, vm_cpu = split_vms(cpu, VMS) row = {"ts": round(t1, 3), "seconds": round(t1 - t0, 3), "load1": load1, "cpu": cpu, "serving_cpu": round(sum(cpu.values()), 3), "aiperf_procs": aiperf()} + if VMS: + row["vm_cpu"] = vm_cpu if SLOT is not None: st1 = load._read("/proc/stat") row.update(slot=SLOT, **slot_fields(st, st1, own)) diff --git a/bench/eatflow/crescent/test_crescent.py b/bench/eatflow/crescent/test_crescent.py index 57e4cf5a..69ae73e6 100644 --- a/bench/eatflow/crescent/test_crescent.py +++ b/bench/eatflow/crescent/test_crescent.py @@ -55,6 +55,34 @@ def test_slot_fields(self): shutil.rmtree(tmp) +class ServingVmTest(unittest.TestCase): + """H62 (EXPERIMENTS.md "H62 pre-registration"): the two KVM guests are this campaign's arms. + SERVING_VMS names their containers; their CPU is recorded as vm_cpu and kept out of the + serving covariate. Without it, every row reads as before.""" + + CPU = {"qwen38-a-hr": 1.25, "qwen38-b-hr": 1.0, "h62-vm-a": 3.5, "h62-vm-b": 2.0} + + def test_vms_split_out_of_the_serving_covariate(self): + serving_cpu, vm_cpu = serving.split_vms(self.CPU, ("h62-vm-a", "h62-vm-b")) + self.assertEqual(serving_cpu, {"qwen38-a-hr": 1.25, "qwen38-b-hr": 1.0}) + self.assertEqual(vm_cpu, {"h62-vm-a": 3.5, "h62-vm-b": 2.0}) + + def test_without_vms_nothing_moves(self): + self.assertEqual(serving.split_vms(self.CPU, ()), (self.CPU, {})) + self.assertEqual(serving.vms_from(""), ()) + self.assertEqual(serving.vms_from("h62-vm-a, h62-vm-b,"), ("h62-vm-a", "h62-vm-b")) + + def test_a_vm_missing_from_the_reading_is_not_invented(self): + serving_cpu, vm_cpu = serving.split_vms({"qwen38-a-hr": 1.0}, ("h62-vm-a",)) + self.assertEqual((serving_cpu, vm_cpu), ({"qwen38-a-hr": 1.0}, {})) + + def test_the_guests_are_not_this_campaigns_by_name(self): + # Another row's hold counts the guests as foreign serving load: their names carry no + # k3dgpu prefix, and hold.sh's cleanup (k3dgpu-e-*, k3dgpu-a-*) never removes them. + for name in ("h62-vm-a", "h62-vm-b"): + self.assertFalse(serving.is_mine(name), name) + + class LockwaitTest(unittest.TestCase): def run_log(self, text, waits=()): import lockwait diff --git a/bench/eatflow/live.sh b/bench/eatflow/live.sh index 28a1dba7..f3c9b5c2 100755 --- a/bench/eatflow/live.sh +++ b/bench/eatflow/live.sh @@ -43,6 +43,7 @@ # solo: ARM = H53, one arm (LIVE_SOLO) per hold on the Arc Pro B70s (bench/eatflow/solo.py); # pairs: ARM = H47, H49, H47b, H49b, H50 or H50o, both arms live in cluster k3dgpu-e-, one hold # of interleaved pairs into LIVE_PAIRS_DIR (LIVE_HOLD = this hold's number); +# ARM = H62: both servers in KVM guests, driven by LIVE_REMOTE_A and LIVE_REMOTE_B (bench/h62); # code: H43 (amendment W1), the coding agent on C12 with `k3d-gpu sandbox up`, one # block per level in LIVE_CODE_LEVELS (default "typical heavy heavy typical") into # LIVE_CODE_DIR/h43-- (shared by an arm's holds, so a level's halves resume); @@ -170,6 +171,7 @@ t0=$(date +%s) el() { echo $(( $(date +%s) - t0 )); } stamp() { echo "== $1 $(date +%T) (+$(el)s)" | tee -a "${OUT}/live.log"; } pids=() +REMOTE=() # H62: the remote arms' control scripts (their port-forwards stop with the hold) # DevUI bearer token for this hold only (the agent and the harness read DEVUI_AUTH_TOKEN). DEVUI_AUTH_TOKEN=$(python3 -c "import secrets; print(secrets.token_hex(16))") export DEVUI_AUTH_TOKEN @@ -180,6 +182,7 @@ cleanup() { for p in "${pids[@]}"; do kill "${p}" 2>/dev/null; done [ -f "${OUT}/mcp.pid" ] && kill "$(cat "${OUT}/mcp.pid")" 2>/dev/null for f in "${OUT}"/agent-*.pid; do [ -f "${f}" ] && kill "$(cat "${f}")" 2>/dev/null; done + for r in "${REMOTE[@]}"; do "${r}" unforward >/dev/null 2>&1; done docker rm -f "${PFX}-litellm" "${PFX}-litellm-a" "${PFX}-litellm-b" "${PFX}-${lc}-srv" "${PFX}-rapl" \ "${PFX}-redis" >/dev/null 2>&1 k3d cluster delete "${K3D_GPU_CLUSTER}" >/dev/null 2>&1 @@ -449,6 +452,7 @@ if [ "${MODE}" = pairs ]; then # What to deploy: the family's arms, or the row's own B arm (H50o: B3). DESIGN=${FAMILY} [ "${ARM}" = H50o ] && DESIGN=H50o + [ "${ARM}" = H62 ] && DESIGN=H62 [ "${ARM}" = H50 ] && KV_GIB=${LIVE_EAT_KV_GIB:-40} # EAT's shipped KV cache (40 GiB) mkdir -p "${PD}" PD=$(cd "${PD}" && pwd) @@ -456,7 +460,52 @@ if [ "${MODE}" = pairs ]; then H47) A=A1 B=A4 ;; H49) A=A1 B=B1; export K3D_GPU_NODE_LABELS=ei-infra-eligible=true,ei-inference-eligible=true ;; H50o) A=A1 B=B3 ;; + H62) A=A1 B=B5 ;; esac + if [ "${DESIGN}" = H62 ]; then + # H62 ("H62 pre-registration"): each server runs in its own KVM guest on this host + # (bench/h62); nothing is deployed here. LIVE_REMOTE_A and LIVE_REMOTE_B name each arm's + # control script (bench/h62/arm.sh ): restart its serving pod, record it, forward + # its Service to a host port and name its pod cgroup for the idle-arm check. From the + # LiteLLM gateways on, the hold is H50's. The local arms below run unchanged in the else + # branch. + RA=${LIVE_REMOTE_A:?H62 needs LIVE_REMOTE_A} RB=${LIVE_REMOTE_B:?H62 needs LIVE_REMOTE_B} + REMOTE=("${RA}" "${RB}") + stamp "restart both servers" + left=$(( CAP - $(el) - 900 )) # ready with at least 900 s of the hold left + "${RA}" restart "${left}" > "${OUT}/restart-a.log" 2>&1 & + ra=$! + "${RB}" restart "${left}" > "${OUT}/restart-b.log" 2>&1 & + rb=$! + wait "${ra}"; xa=$? + wait "${rb}"; xb=$? + if [ "${xa}" != 0 ] || [ "${xb}" != 0 ]; then + "${RA}" logs "${OUT}/remote-a" > /dev/null 2>&1 + "${RB}" logs "${OUT}/remote-b" > /dev/null 2>&1 + if [ "${xa}" = 3 ] || [ "${xb}" = 3 ]; then + echo "NOT DELIVERABLE: not ready with 900 s of the hold left (restart A ${xa}, B ${xb})" | tee -a "${OUT}/live.log" + exit 3 + fi + echo "restart failed (A ${xa}, B ${xb})" | tee -a "${OUT}/live.log" + exit 1 + fi + stamp ready + "${RA}" records "${OUT}/remote-a" > "${OUT}/records-a.log" 2>&1 || { echo "A records failed"; exit 1; } + "${RB}" records "${OUT}/remote-b" > "${OUT}/records-b.log" 2>&1 || { echo "B records failed"; exit 1; } + "${RA}" forward >> "${OUT}/live.log" 2>&1 || { echo "A forward failed"; exit 1; } + "${RB}" forward >> "${OUT}/live.log" 2>&1 || { echo "B forward failed"; exit 1; } + P_SRV=$("${RA}" port) BPORT=$("${RB}" port) API_B=/v1 + cg_a=$("${RA}" cgroup) || { echo "no cgroup for the A pod"; exit 1; } + cg_b=$("${RB}" cgroup) || { echo "no cgroup for the B pod"; exit 1; } + echo "pod cgroups: ${A} ${cg_a}; ${B} ${cg_b}" | tee -a "${OUT}/live.log" + python3 "${HERE}/pairs.py" usage "${cg_a}" "${cg_b}" >> "${OUT}/live.log" 2>&1 || { echo "pod cpu.stat unreadable"; exit 1; } + wait_http "http://127.0.0.1:${P_SRV}/v1/models" 120 || { echo "A server not answering"; exit 3; } + curl -fsS "http://127.0.0.1:${P_SRV}/v1/models" > "${OUT}/models-a.json" + wait_http "http://127.0.0.1:${BPORT}${API_B}/models" 300 || { echo "B server not answering"; exit 3; } + curl -fsS "http://127.0.0.1:${BPORT}${API_B}/models" > "${OUT}/models-b.json" + SERVED_A=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['data'][0]['id'])" "${OUT}/models-a.json") + SERVED_B=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['data'][0]['id'])" "${OUT}/models-b.json") + else # The B arm's server (every design but H47's one pod): namespace, Deployment, Service, # service port and OpenAI path. B_NS=default B_DEP=${B_REL} B_SVC=${B_REL}-service B_SPORT=80 B_API=/v1 @@ -590,6 +639,7 @@ if [ "${MODE}" = pairs ]; then curl -fsS "http://127.0.0.1:${BPORT}${API_B}/models" > "${OUT}/models-b.json" SERVED_B=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['data'][0]['id'])" "${OUT}/models-b.json") fi + fi litellm() { # litellm x served port llm_port api_path printf 'model_list:\n - model_name: %s\n litellm_params:\n model: openai/%s\n api_base: http://127.0.0.1:%s%s\n api_key: not-needed\n timeout: 600\n' \ "${ALIAS}" "$2" "$3" "$5" > "${OUT}/litellm-$1.yaml" diff --git a/bench/eatflow/pairs.py b/bench/eatflow/pairs.py index 723f8452..d58c648a 100644 --- a/bench/eatflow/pairs.py +++ b/bench/eatflow/pairs.py @@ -17,12 +17,16 @@ pairs' pre-windows. bg_cpu, package W and throttle during the attempt are reported, never gated. -Load rule L3 (H50, H50o, H53; EXPERIMENTS.md "Crescent amendment A3"): the same +Load rule L3 (H50, H50o, H53, H62; EXPERIMENTS.md "Crescent amendment A3"): the same pre-windows, no match filter. Every complete pair enters the statistic; a balance gate (a sign-flip test of arm on each pre covariate, p < 0.01 gates) replaces it; a two-regressor adjustment and L2's matched subset are reported, never gated. The arm order of pair j is drawn from the registered seed (order), and each pairs directory carries one registration.json (register). Python 3 standard library only. + +A pod cgroup is a local cgroup v2 directory, or "exec:" for a pod on another +machine (H62, EXPERIMENTS.md "H62 pre-registration": each server runs in its own KVM guest): +the command prints that pod's cpu.stat, and the idle-arm check reads it like a local one. """ import glob import json @@ -103,9 +107,30 @@ def total_jiffies(stat_text, cpus): return tot +EXEC = "exec:" # a pod cgroup read through a command (H62: a pod in a KVM guest) +EXEC_TIMEOUT_S = 15.0 + + +def _exec_text(cmdline): + """stdout of a command line (shlex-split, no shell), or "" when it fails or times out.""" + argv = shlex.split(cmdline) + if not argv: + return "" + try: + p = subprocess.run(argv, capture_output=True, text=True, timeout=EXEC_TIMEOUT_S) + except (OSError, subprocess.SubprocessError): + return "" + return p.stdout if p.returncode == 0 else "" + + def cgroup_usage_usec(path): - """cpu.stat usage_usec of a cgroup v2 directory (None when unreadable).""" - for line in _read(os.path.join(path, "cpu.stat")).splitlines(): + """cpu.stat usage_usec of a cgroup v2 directory, or of "exec:", a command that + prints a pod's cpu.stat on another machine (None when unreadable).""" + if path.startswith(EXEC): + text = _exec_text(path[len(EXEC):]) + else: + text = _read(os.path.join(path, "cpu.stat")) + for line in text.splitlines(): k, _, v = line.partition(" ") if k == "usage_usec" and v.strip().isdigit(): return int(v) @@ -512,13 +537,14 @@ def boot_share(d, pred, boot=stats.BOOT, seed=stats.SEED, stat=None): # Rule L3 rows: the seed of their arm order, fixed in the pre-registration (amendment A3). ORDER_SEEDS = {"H50": "H50-crescent-2026-10-20", "H50o": "H50o-crescent-2026-10-20", - "H53": "H53-crescent-2026-10-20"} + "H53": "H53-crescent-2026-10-20", "H62": "H62-crescent-2026-10-29"} # Hypothesis id -> (design, load rule). H47b and H49b are H47 and H49 under rule L2; H50 is # H49b's design on the Xeon 6741P, H50o the same with arm B3 (EXPERIMENTS.md "H50-H52 pre-registration"); -# H53 the H49 statistic over solo-hold block pairs on the Arc Pro B70s (solo.py; block-level verdict, "H53 pre-registration"). +# H53 the H49 statistic over solo-hold block pairs on the Arc Pro B70s (solo.py; block-level verdict, "H53 pre-registration"); +# H62 H50's design with arm B5, Intel's EAS default, each server in a KVM guest ("H62 pre-registration"). DESIGNS = {"H47": ("H47", "L1"), "H49": ("H49", "L1"), "H47b": ("H47", "L2"), "H49b": ("H49", "L2"), - "H50": ("H49", "L3"), "H50o": ("H49", "L3"), "H53": ("H49", "L3")} + "H50": ("H49", "L3"), "H50o": ("H49", "L3"), "H53": ("H49", "L3"), "H62": ("H49", "L3")} # The ledger a rule-L3 row must be pre-registered in before any of its holds runs. @@ -814,13 +840,20 @@ def analyze(attempts, pairs, hyp, a, b, holds=None, registration=None, solo=Fals if __name__ == "__main__": # `pairs.py design HYP`: " " (live.sh reads the rule from here, DESIGNS only). # `pairs.py arm HYP N`: the arm solo hold N must run (solo-hold.sh checks it before the lock). + # `pairs.py usage SPEC...`: each pod cgroup's usage_usec, read once; exit 1 if one is unreadable + # (live.sh checks the H62 arms' exec: specs with it before a hold). try: if len(sys.argv) == 3 and sys.argv[1] == "design": print(*design(sys.argv[2])) elif len(sys.argv) == 4 and sys.argv[1] == "arm": design(sys.argv[2]) print(expected_arm(sys.argv[2], int(sys.argv[3]))) + elif len(sys.argv) >= 3 and sys.argv[1] == "usage": + got = [(spec, cgroup_usage_usec(spec)) for spec in sys.argv[2:]] + for spec, v in got: + print(f"usage_usec {v} {spec}") + sys.exit(1 if any(v is None for _, v in got) else 0) else: - raise SystemExit("usage: pairs.py design HYP | pairs.py arm HYP N") + raise SystemExit("usage: pairs.py design HYP | pairs.py arm HYP N | pairs.py usage SPEC...") except (ValueError, KeyError) as e: raise SystemExit(f"pairs.py: {e!r}") diff --git a/bench/eatflow/test_pairs.py b/bench/eatflow/test_pairs.py index 09a525cb..6b062708 100644 --- a/bench/eatflow/test_pairs.py +++ b/bench/eatflow/test_pairs.py @@ -931,6 +931,84 @@ def test_h50_and_h51_run_eat_release_110_b1(self): self.assertIn("EAT_PR21=c86c42b073ee533e9fb9e9b9330b9d1478e4bf47", self.read("crescent", "setup.sh")) +class H62Test(unittest.TestCase): + """H62 (EXPERIMENTS.md "H62 pre-registration"): H50's design, statistic and rule (L3) with + arm B5, Intel's EAS default, and both servers in KVM guests. A pod cgroup can be read + through a command ("exec:"), so the idle-arm check reads the pod inside its guest.""" + + REG = {"hyp": "H62", "rule": "L3", "seed": "H62-crescent-2026-10-29"} + + def read_live(self): + with open(os.path.join(HERE, "live.sh")) as f: + return f.read() + + def test_h62_is_h50s_design_and_rule_with_its_own_seed(self): + self.assertEqual(PR.design("H62"), ("H49", "L3")) + self.assertEqual(PR.ORDER_SEEDS["H62"], self.REG["seed"]) + self.assertIn("H62", AB.PAIRS_HYPS) + at = [dict(r, arm="B5") if r["arm"] == "B1" else r for r in L2Test.h49_like(L2Test())] + res = PR.analyze(at, pairs_of(24), "H62", "A1", "B5", registration=self.REG) + ref = PR.analyze(L2Test.h49_like(L2Test()), pairs_of(24), "H50", "A1", "B1", + registration={"hyp": "H50", "rule": "L3", "seed": PR.ORDER_SEEDS["H50"]}) + self.assertEqual((res["rule"], res["arms"], res["verdict"]), ("L3", ["A1", "B5"], "SUPPORTED")) + self.assertEqual(res["paired_diff"], ref["paired_diff"]) # H50's statistic, unchanged + self.assertEqual(res["confidence"]["support_share"], ref["confidence"]["support_share"]) + with self.assertRaises(ValueError): # H50's registration is not H62's + PR.analyze(at, pairs_of(24), "H62", "A1", "B5", + registration={"hyp": "H50", "rule": "L3", "seed": PR.ORDER_SEEDS["H50"]}) + + def test_exec_cgroup_reads_a_command(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + fake = os.path.join(tmp, "arm.sh") + with open(fake, "w") as f: + f.write('#!/bin/sh\n[ "$1 $2 $3" = "A1 cpustat /sys/fs/cgroup/x" ] || exit 7\n' + 'printf "usage_usec 9001\\nuser_usec 5\\n"\n') + os.chmod(fake, 0o755) + self.assertEqual(PR.cgroup_usage_usec(f"exec:{fake} A1 cpustat /sys/fs/cgroup/x"), 9001) + self.assertIsNone(PR.cgroup_usage_usec(f"exec:{fake} A1 cpustat /elsewhere")) # non-zero exit + self.assertIsNone(PR.cgroup_usage_usec(f"exec:{os.path.join(tmp, 'missing')}")) + self.assertIsNone(PR.cgroup_usage_usec("exec:")) + # A probe snapshot carries the remote reading like a local one; the window uses it. + probe = PR.Probe({"A1": f"exec:{fake} A1 cpustat /sys/fs/cgroup/x"}, [], rapl=False) + self.assertEqual(list(probe.snap()["pods"].values()), [9001]) + # `pairs.py usage SPEC...` reads each spec once: what live.sh checks before a hold. + import subprocess + ok = subprocess.run([sys.executable, os.path.join(HERE, "pairs.py"), "usage", + f"exec:{fake} A1 cpustat /sys/fs/cgroup/x"], capture_output=True, text=True) + self.assertEqual(ok.returncode, 0, ok.stderr) + self.assertIn("9001", ok.stdout) + bad = subprocess.run([sys.executable, os.path.join(HERE, "pairs.py"), "usage", + f"exec:{fake} A1 cpustat /sys/fs/cgroup/x", f"exec:{fake} B5 cpustat /y"], + capture_output=True, text=True) + self.assertEqual(bad.returncode, 1) + + def test_live_sh_runs_h62_against_remote_arms(self): + live = self.read_live() + self.assertIn('[ "${ARM}" = H62 ] && DESIGN=H62', live) + self.assertRegex(live, r"\n\s*H62\) A=A1 B=B5 ;;") + pairs = live[live.index('if [ "${MODE}" = pairs ]; then'):] + remote = pairs[pairs.index('if [ "${DESIGN}" = H62 ]; then'):pairs.index("\n else\n")] + # Neither arm is deployed on this host: no cluster, no staging, no local port-forward. + for local in ("cluster_up", "stage_path", "kubectl ", "agent manifest", "helm template"): + self.assertNotIn(local, remote, local) + for step in ('"${RA}" restart "${left}"', '"${RB}" restart "${left}"', 'records "${OUT}/remote-a"', + 'records "${OUT}/remote-b"', '"${RA}" forward', '"${RB}" forward', 'P_SRV=$("${RA}" port)', + 'BPORT=$("${RB}" port)', 'cg_a=$("${RA}" cgroup)', 'cg_b=$("${RB}" cgroup)', + 'python3 "${HERE}/pairs.py" usage "${cg_a}" "${cg_b}"'): + self.assertIn(step, remote, step) + self.assertIn("left=$(( CAP - $(el) - 900 ))", remote) # the 900 s readiness rule + self.assertIn("NOT DELIVERABLE: not ready with 900 s of the hold left", remote) + # The local arms run in the else branch unchanged, and the harness after it is shared. + local = pairs[pairs.index("\n else\n"):pairs.index(" litellm() {")] + self.assertIn(" cluster_up\n", local) + self.assertIn("\n fi\n", local[-40:]) + # The guests' port-forwards stop with the hold. + clean = live[live.index("cleanup() {"):live.index("trap cleanup EXIT")] + self.assertIn('for r in "${REMOTE[@]}"; do "${r}" unforward >/dev/null 2>&1; done', clean) + self.assertIn("REMOTE=()", live[:live.index("cleanup() {")]) + + class B3RecordsTest(unittest.TestCase): """H50o amendment O1 item 2: the records of B3's pulled model that the pre-registration names (the init container's log, every file's sha256, the graph each hold serves), read from the diff --git a/bench/h62/arm-A1 b/bench/h62/arm-A1 new file mode 100755 index 00000000..3bac3c89 --- /dev/null +++ b/bench/h62/arm-A1 @@ -0,0 +1,3 @@ +#!/usr/bin/env bash +# arm-A1: bench/h62/arm.sh for arm A1, as one word (live.sh's LIVE_REMOTE_A). +exec "$(dirname "$0")/arm.sh" A1 "$@" diff --git a/bench/h62/arm-B5 b/bench/h62/arm-B5 new file mode 100755 index 00000000..5b9c4b02 --- /dev/null +++ b/bench/h62/arm-B5 @@ -0,0 +1,3 @@ +#!/usr/bin/env bash +# arm-B5: bench/h62/arm.sh for arm B5, as one word (live.sh's LIVE_REMOTE_B). +exec "$(dirname "$0")/arm.sh" B5 "$@" diff --git a/bench/h62/arm.sh b/bench/h62/arm.sh new file mode 100755 index 00000000..4ee85b17 --- /dev/null +++ b/bench/h62/arm.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# arm.sh ARM CMD [ARG]: one H62 arm's server in its KVM guest (EXPERIMENTS.md "H62 +# pre-registration"). bench/eatflow/live.sh's remote-arm mode calls it through arm-A1 and arm-B5. +# port the host port its Service is forwarded to (the guest's hostfwd) +# restart SECONDS delete its serving pod, then wait until a new pod is Ready; exit 3 when none is +# Ready within SECONDS (the hold's 900 s readiness rule passes the time left) +# records DIR the pod, the CPU ISA it sees (amx_tile lines), its cpuset and nproc, the nodes, +# the guest's load and memory, and B5's LLMInferenceService +# logs DIR every pod, events, the pod description and the server log (a hold that fails) +# forward (re)start `kubectl port-forward` of its Service in the guest, on the serve port +# unforward stop that port-forward +# cgroup "exec: ARM cpustat ": its pod's cgroup in the guest, as the +# idle-arm check reads it (bench/eatflow/pairs.py) +# cpustat DIR that cgroup's cpu.stat +# ARM: A1 (VM a: k3d-gpu's agent OVMS) or B5 (VM b: EAS's gemma-4-26b LLMInferenceService). +# H62_STATE (default ~/k3dgpu-bench/h62) holds ssh_config; H62_SSH is the ssh client (tests stub +# it); H62_POLL_S is the restart poll period (default 5 s). Each guest's ~/h62/env sets PATH and +# KUBECONFIG for kubectl there. +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +SELF=${HERE}/arm.sh +P() { python3 "${HERE}/h62.py" "$@"; } +ARM=${1:?usage: arm.sh A1|B5 CMD [ARG]} CMD=${2:?usage: arm.sh A1|B5 CMD [ARG]} +fields=$(P arm-fields "${ARM}" 2>/dev/null) || { echo "arm.sh: no arm ${ARM} (A1 or B5)" >&2; exit 2; } +read -r _ HOST PORT NS SEL SVC SPORT CTR EXTRA <<< "${fields}" +STATE=${H62_STATE:-${HOME}/k3dgpu-bench/h62} +# g CMD: one command in the guest, after its ~/h62/env. +g() { ${H62_SSH:-ssh} -F "${STATE}/ssh_config" "${HOST}" ". ~/h62/env && $*"; } +pods() { g "kubectl -n ${NS} get pods -l ${SEL} -o json"; } +ready_pod() { pods | P pick-pod "${1:-}"; } # " " of a Ready pod whose uid is not in $1 +case ${CMD} in +port) + echo "${PORT}" ;; +restart) + left=${3:-} + [ -n "${left}" ] || { echo "usage: arm.sh ${ARM} restart SECONDS" >&2; exit 2; } + [ "${left}" -gt 0 ] 2>/dev/null || { echo "no time left to restart"; exit 3; } + end=$(( SECONDS + left )) + old=$(pods | P uids) || { echo "cannot list the pods"; exit 1; } + g "kubectl -n ${NS} delete pod -l ${SEL} --wait=false" || { echo "cannot delete the pods"; exit 1; } + while [ "${SECONDS}" -lt "${end}" ]; do + if p=$(ready_pod "${old}"); then echo "ready: ${p}"; exit 0; fi + sleep "${H62_POLL_S:-5}" + done + echo "not ready within ${left} s" + exit 3 ;; +records) + d=${3:?usage: arm.sh ARM records DIR} + mkdir -p "${d}" + read -r name uid <<< "$(ready_pod "")" + [ -n "${name:-}" ] || { echo "no Ready pod"; exit 1; } + echo "${name} ${uid}" > "${d}/pod.txt" + g "kubectl -n ${NS} get pod ${name} -o json" > "${d}/pod.json" + g "kubectl -n ${NS} exec ${name} -c ${CTR} -- grep -c amx_tile /proc/cpuinfo" > "${d}/isa.txt" 2>&1 + g "kubectl -n ${NS} exec ${name} -c ${CTR} -- sh -c 'cat /sys/fs/cgroup/cpuset.cpus.effective; nproc'" \ + > "${d}/cpuset.txt" 2>&1 + g "kubectl get nodes -o json" > "${d}/nodes.json" + g "uptime; free -g; grep -c amx_tile /proc/cpuinfo; nproc" > "${d}/guest.txt" 2>&1 + [ -z "${EXTRA}" ] || g "kubectl -n ${NS} get ${EXTRA} -o yaml" > "${d}/extra.yaml" 2>&1 + g "kubectl get pods -A -o wide" > "${d}/pods.txt" 2>&1 + echo "records: ${d}" ;; +logs) + d=${3:?usage: arm.sh ARM logs DIR} + mkdir -p "${d}" + g "kubectl get pods -A -o wide" > "${d}/pods.txt" 2>&1 + g "kubectl get events -A --sort-by=.lastTimestamp" > "${d}/events.txt" 2>&1 + g "kubectl -n ${NS} describe pods -l ${SEL}" > "${d}/describe.txt" 2>&1 + g "kubectl -n ${NS} logs -l ${SEL} -c ${CTR} --tail=1000" > "${d}/server.log" 2>&1 + echo "logs: ${d}" ;; +forward) + g "pid=\$(cat ~/h62/pf-${ARM}.pid 2>/dev/null) && kill \${pid} 2>/dev/null;" \ + "setsid nohup kubectl -n ${NS} port-forward --address 0.0.0.0 ${SVC} ${PORT}:${SPORT}" \ + "> ~/h62/pf-${ARM}.log 2>&1 < /dev/null & echo \$! > ~/h62/pf-${ARM}.pid" ;; +unforward) + g "pid=\$(cat ~/h62/pf-${ARM}.pid 2>/dev/null) && kill \${pid} 2>/dev/null; rm -f ~/h62/pf-${ARM}.pid" ;; +cgroup) + read -r _ uid <<< "$(ready_pod "")" + [ -n "${uid:-}" ] || { echo "no Ready pod" >&2; exit 1; } + # cgroupfs driver (k3s in VM a's k3d node): pod; systemd driver (Kubespray's containerd + # in VM b): ...-pod.slice. + dir=$(g "find /sys/fs/cgroup -maxdepth 9 -type d \\( -name pod${uid} -o -name '*pod${uid//-/_}.slice' \\) -print -quit") + [ -n "${dir}" ] || { echo "no cgroup for pod ${uid}" >&2; exit 1; } + echo "exec:${SELF} ${ARM} cpustat ${dir}" ;; +cpustat) + g "cat ${3:?usage: arm.sh ARM cpustat DIR}/cpu.stat" ;; +*) + echo "usage: arm.sh A1|B5 port|restart SECONDS|records DIR|logs DIR|forward|unforward|cgroup|cpustat DIR" >&2 + exit 2 ;; +esac diff --git a/bench/h62/campaign.sh b/bench/h62/campaign.sh new file mode 100755 index 00000000..5f7f7034 --- /dev/null +++ b/bench/h62/campaign.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# campaign.sh: H62 end to end on crescent (EXPERIMENTS.md "H62 pre-registration"), from the H62 +# checkout at the deploy commit: setup.sh, smoke.sh, then holds (pairs-hold.sh 1, 2, ...) until +# B24's 24 pairs are recorded or 8 holds have run, then the score. teardown.sh runs at the end, +# also after an abort. +# - setup exit 6 (the host gate still failed after 60 min): the two qwen38 serving pairs are +# stopped under the box owner's grant (gpu-serving.sh stop, never rm) and setup runs once more; +# teardown.sh starts them again +# - a hold that exits 3 (a server not ready with 900 s left) counts as one of the 8 and runs no +# pair; after 2 such holds in a row no further hold runs (INCONCLUSIVE, not deliverable) +# Records: ~/k3dgpu-bench/results/H62 (campaign.log, setup/, pairs/, hold/, report.txt, vmtax.json). +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +B=${HOME}/k3dgpu-bench +R=${B}/results/H62 +MAX_HOLDS=8 NOT_DELIVERABLE_MAX=2 +mkdir -p "${R}" +log() { echo "== campaign $* $(date -u +%FT%TZ)" | tee -a "${R}/campaign.log"; } +teardown() { bash "${HERE}/teardown.sh" >> "${R}/teardown.log" 2>&1; log "teardown exit $?"; } +trap teardown EXIT +log "start at $(git -C "${REPO}" rev-parse HEAD)" +bash "${HERE}/setup.sh" +rc=$? +if [ "${rc}" = 6 ]; then + log "host gate failed: stopping the qwen38 serving pairs (owner's grant; docker stop, never rm)" + for c in qwen38-a-hr qwen38-b-hr; do + if bash "${REPO}/bench/eatflow/crescent/gpu-serving.sh" stop "${c}" >> "${R}/serving.log" 2>&1; then + echo "${c}" >> "${R}/serving-stopped.txt" + fi + done + bash "${HERE}/setup.sh" + rc=$? +fi +[ "${rc}" = 0 ] || { log "setup exit ${rc}: no hold"; exit "${rc}"; } +bash "${HERE}/smoke.sh" +rc=$? +[ "${rc}" = 0 ] || { log "wiring smoke failed (exit ${rc}): no hold"; exit 9; } +nd=0 +for n in $(seq 1 "${MAX_HOLDS}"); do + left=$(python3 "${HERE}/h62.py" pairs-left "${R}/pairs") + [ "${left}" -gt 0 ] || break + log "hold ${n}: ${left} pair(s) left" + bash "${HERE}/pairs-hold.sh" "${n}" + rc=$? + log "hold ${n} exit ${rc}" + if [ "${rc}" = 3 ]; then + nd=$(( nd + 1 )) + [ "${nd}" -lt "${NOT_DELIVERABLE_MAX}" ] || { log "${nd} holds in a row not deliverable: no further hold"; break; } + else + nd=0 + fi +done +python3 "${REPO}/bench/eatflow/agentbench.py" pairs-report "${R}/pairs" --hyp H62 --arm-a A1 --arm-b B5 \ + > "${R}/report.txt" 2>&1 +log "pairs-report exit $?" +h50=${REPO}/bench/eatflow/results/2026-10-20/H50/pairs/analysis.json +if [ -f "${R}/pairs/analysis.json" ]; then + python3 "${HERE}/h62.py" vmtax "${h50}" "${R}/pairs/analysis.json" > "${R}/vmtax.json" +fi diff --git a/bench/h62/guest-a.sh b/bench/h62/guest-a.sh new file mode 100755 index 00000000..7388701c --- /dev/null +++ b/bench/h62/guest-a.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# guest-a.sh [deploy]: VM a's setup for H62's A1 (EXPERIMENTS.md "H62 pre-registration"), run in +# the guest by setup.sh as user ubuntu, under its 45 min ceiling. A1 is H50's A1 unchanged: +# 1. Docker CE from Docker's apt repository at the pinned version (its key's fingerprint +# checked first) and python3-yaml for bench/apb/stage.py; then this script runs again in a +# shell with the docker group, as `deploy` +# 2. A1's IR from the host's read-only share (virtfs tag h62share), checked against its +# manifest.json +# 3. `k3d-gpu up` at the deploy commit (~/h62/k3d-gpu, copied by setup.sh), with H50's node image +# and crescent amendment C1's kubelet eviction thresholds, cluster k3dgpu-e-h62 +# 4. `k3d-gpu agent manifest` with live.sh's gemma4 settings for A1 (CPU, the snapped QAT IR, +# tool parser gemma4, no reasoning parser, no draft, no autoquant), staged and applied as +# live.sh does, and rolled out +# 5. records: amendment A4's `k3d-gpu plan` line, versions, the node, the pod +# k3d and kubectl are setup.sh's checked binaries in ~/h62/bin. Records: ~/h62/records. +set -euo pipefail +G=${HOME}/h62 +P() { python3 "${G}/h62.py" "$@"; } +R=${G}/records +REPO=${G}/k3d-gpu +mkdir -p "${R}" +export PATH=${G}/bin:${PATH} KUBECONFIG=${G}/kubeconfig XDG_STATE_HOME=${G}/state K3D_GPU_CLUSTER=k3dgpu-e-h62 +step() { echo "== $1 $(date -u +%FT%TZ)" | tee -a "${R}/steps.txt"; } +if [ "${1:-}" != deploy ]; then + step docker + # ${PATH} expands when a command in the guest sources ~/h62/env (bench/h62/arm.sh). + # shellcheck disable=SC2016 + printf 'export PATH=%s/bin:${PATH} KUBECONFIG=%s/kubeconfig\n' "${G}" "${G}" > "${G}/env" + curl -fsSLo "${G}/docker.asc" https://download.docker.com/linux/ubuntu/gpg + gpg --show-keys --with-colons "${G}/docker.asc" | awk -F: '/^fpr/ {print $10}' | grep -x "$(P pin docker_gpg_fpr)" > /dev/null \ + || { echo "Docker's apt key is not $(P pin docker_gpg_fpr)"; exit 1; } + sudo install -m 0755 -d /etc/apt/keyrings + sudo install -m 0644 "${G}/docker.asc" /etc/apt/keyrings/docker.asc + echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable" \ + | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + sudo apt-get update -q 2>&1 | tee "${R}/apt-update.log" > /dev/null + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q \ + docker-ce="$(P pin docker_ce)" docker-ce-cli="$(P pin docker_ce)" containerd.io="$(P pin containerd)" \ + python3-yaml="$(P pin python3_yaml)" 2>&1 | tee "${R}/apt-install.log" > /dev/null + sudo usermod -aG docker "$(id -un)" + exec sg docker -c "bash ${G}/guest-a.sh deploy" +fi +step ir +sudo mkdir -p /mnt/h62-ir +mountpoint -q /mnt/h62-ir || sudo mount -t 9p -o trans=virtio,version=9p2000.L,ro h62share /mnt/h62-ir +P ir-verify /mnt/h62-ir > "${R}/ir-verify.txt" +step up +K3D_GPU_IMAGE="$(P pin node_image)" \ + K3D_GPU_K3S_ARGS="--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1%" \ + timeout 900 "${REPO}/scripts/k3d-gpu" up > "${R}/up.log" 2>&1 +step manifest +K3D_GPU_AGENT_DEVICE=CPU K3D_GPU_AGENT_MODEL=k3dgpu/gemma-4-26B-A4B-it-qat-int4-ov K3D_GPU_AGENT_MODEL_NAME=gemma-4-26b-a4b-it \ + K3D_GPU_AGENT_DRAFT=off K3D_GPU_AGENT_TOOL_PARSER=gemma4 K3D_GPU_AGENT_REASONING_PARSER='' K3D_GPU_AUTOQUANT=off \ + "${REPO}/scripts/k3d-gpu" agent manifest > "${R}/agent.yaml" 2> "${R}/agent.err" +python3 "${REPO}/bench/apb/stage.py" ovms "${R}/agent.yaml" "${R}/deploy-a.yaml" +step stage +# As live.sh's stage_path: the IR copied into the node at the path the staged manifest mounts. +node=$(docker ps --filter "label=k3d.cluster=${K3D_GPU_CLUSTER}" --filter label=k3d.role=server --format '{{.Names}}' | head -1) +[ -n "${node}" ] || { echo "no k3d server node"; exit 1; } +t=/var/apb-models/k3dgpu/gemma-4-26B-A4B-it-qat-int4-ov +docker exec "${node}" mkdir -p "${t}" /var/apb-models/.ov_cache +docker cp -L /mnt/h62-ir/. "${node}:${t}/" +docker exec "${node}" rm -rf "${t}/.cache" "${t}/.git" +docker exec "${node}" chown -R 5000:5000 "${t}" +docker exec "${node}" chown 5000:5000 /var/apb-models/.ov_cache +docker exec "${node}" chmod 1777 /var/apb-models +step apply +kubectl apply -f "${R}/deploy-a.yaml" > "${R}/apply.log" +kubectl -n k3d-gpu-agent rollout status deploy/ovms --timeout=1500s > "${R}/rollout.log" 2>&1 +step records +{ timeout 600 "${REPO}/scripts/k3d-gpu" plan --device cpu --model google/gemma-4-26B-A4B-it --explain; echo "rc=$?"; } \ + > "${R}/plan.txt" 2>&1 || true +{ docker version; k3d version; kubectl version --client; uname -r; } > "${R}/versions.txt" 2>&1 +kubectl get nodes -o json > "${R}/nodes.json" +kubectl -n k3d-gpu-agent get pods -o json > "${R}/pods.json" +docker exec "${node}" sh -c "cd ${t} && sha256sum ./*.xml ./*.bin" > "${R}/staged-ir.txt" +step finished diff --git a/bench/h62/guest-b.sh b/bench/h62/guest-b.sh new file mode 100755 index 00000000..2fb70ad0 --- /dev/null +++ b/bench/h62/guest-b.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# guest-b.sh: VM b's setup for H62's B5 (EXPERIMENTS.md "H62 pre-registration"), run in the guest +# by setup.sh as user ubuntu, under its 120 min ceiling. B5 is Intel's documented single-node +# install of the Agent Toolkit: EAT release-2.0.0 docs/single-node-deployment.md from Step 1, with +# the clone the v1.2.0-rc.1 release notes give in place of the guide's `-b release-1.2`, on the +# inventory EAS ships (Option A, localhost: Kubespray installs Kubernetes here). +# - prerequisites.md "Node Kernel Settings", verbatim +# - clone, D1 (the layer manifest's two branch revs pinned to the commits Intel's comments on +# those lines name, before init), configure, init agent-toolkit +# - the documented edit (gemma-4-26b in llm_services_autodeploy_models) and D2 (gemma-4-26b's +# model_id is the QAT checkpoint), then install agent-toolkit +# - wait for the gemma-4-26b workload pod, then the records +# Nothing else is changed: guardrail, hybrid router, observability, NRI balloons, Istio, MetalLB +# and the agent sandbox are EAS's and EAT's defaults. Records: ~/h62/records (no kubeconfig, no +# secret file). +set -euo pipefail +G=${HOME}/h62 +P() { python3 "${G}/h62.py" "$@"; } +R=${G}/records +E=${HOME}/enterprise-ai-solutions +TAG=$(P pin eas_tag) SHA=$(P pin eas_sha) +mkdir -p "${R}" +step() { echo "== $1 $(date -u +%FT%TZ)" | tee -a "${R}/steps.txt"; } +echo "HF_TOKEN: $([ -n "${HF_TOKEN:-}" ] && echo set || echo unset)" > "${R}/hf-token.txt" +step sysctl +sudo tee /etc/sysctl.d/90-inotify.conf > /dev/null <<'EOF' +fs.inotify.max_user_instances = 8192 +fs.inotify.max_user_watches = 1048576 +EOF +sudo sysctl --system 2>&1 | tee "${R}/sysctl.txt" > /dev/null +step clone +[ -d "${E}" ] || git clone -q -b "${TAG}" https://github.com/intel/enterprise-ai-solutions.git "${E}" +cd "${E}" +[ "$(git rev-parse HEAD)" = "${SHA}" ] || { echo "EAS ${TAG} is not ${SHA}"; exit 1; } +step d1 +python3 "${G}/h62.py" eas-pin configs/repos/repos.agent-toolkit.yaml > "${R}/d1.diff" +step configure +# --force answers configure's one y/N prompt (es_auto_installer.sh usage: "required in CI, where +# there is no terminal"); configure's own steps are unchanged. +./es_auto_installer.sh configure --force > "${R}/configure.log" 2>&1 +step init +./es_auto_installer.sh init agent-toolkit > "${R}/init.log" 2>&1 +for pair in "enterprise.ai-inference $(P pin ei_sha)" "enterprise.agent-toolkit $(P pin eat_sha)"; do + read -r dest want <<< "${pair}" + have=$(git -C "ext/${dest}" rev-parse HEAD) + echo "${dest} ${have}" >> "${R}/ext-commits.txt" + [ "${have}" = "${want}" ] || { echo "ext/${dest} is at ${have}, not ${want}"; exit 1; } +done +step edit +# base_domain_name stays EAS's default: global_config.yaml is not edited +python3 "${G}/h62.py" eas-edit env/local > "${R}/edit.diff" +for f in global_config.yaml config.agent-toolkit.yaml config.inference.yaml models.yaml inventory/hosts.yaml; do + [ ! -f "env/local/${f}" ] || cp "env/local/${f}" "${R}/env-${f//\//-}" +done +step install +./es_auto_installer.sh install agent-toolkit > "${R}/install.log" 2>&1 +step ready +export KUBECONFIG=${E}/env/local/kubeconfig.yaml +printf 'export KUBECONFIG=%s\n' "${KUBECONFIG}" > "${G}/env" +NS=$(P arm B5 ns) SEL=$(P arm B5 selector) +until kubectl -n "${NS}" get pods -l "${SEL}" -o json | P pick-pod "" > "${R}/ready-pod.txt"; do sleep 15; done +step records +{ ./es_auto_installer.sh status; echo "rc=$?"; } > "${R}/status.txt" 2>&1 || true +{ python3 --version; yq --version; kubectl version; helm version; uname -r; } > "${R}/versions.txt" 2>&1 || true +kubectl get nodes -o json > "${R}/nodes.json" +kubectl get pods -A -o wide > "${R}/pods.txt" +kubectl -n "${NS}" get pods -o json > "${R}/llm-inference-pods.json" +kubectl -n "${NS}" get llminferenceservices -o yaml > "${R}/llminferenceservices.yaml" +step finished diff --git a/bench/h62/h62.py b/bench/h62/h62.py new file mode 100644 index 00000000..88ced803 --- /dev/null +++ b/bench/h62/h62.py @@ -0,0 +1,528 @@ +#!/usr/bin/env python3 +"""h62.py: the pure parts of the H62 runner (EXPERIMENTS.md "H62 pre-registration"): k3d-gpu's +CPU stack A1 against B5, Intel's documented single-node default (EAS v1.2.0-rc.1 installing +EAT 2.0), each in its own KVM guest on crescent. Python 3 standard library only, so the same +file runs on the host and in either guest. + + h62.py pin NAME one pin of PINS (the shell scripts read every pin here) + h62.py vm VM FIELD a field of VMS (VM a or b) + h62.py arm ARM FIELD a field of ARMS (A1 or B5) + h62.py arm-fields ARM every field arm.sh reads, on one line + h62.py ssh-config STATE the ssh_config of both guests + h62.py user-data VM PUBKEY_FILE the guest's cloud-init user-data + h62.py meta-data VM its cloud-init meta-data + h62.py qemu-argv VM STATE IMAGE UID GID KVM_GID [SHARE] + the `docker run` of the guest's QEMU, one argument per line + h62.py host-facts the host gate's facts (MemAvailable, / used, reachability), JSON + h62.py gates PHASE FACTS_JSON PASS/FAIL lines of one gate phase (host, guest, nfd, + weights); exit 1 on any FAIL + h62.py ir-verify DIR A1's IR against its manifest.json; exit 1 on any mismatch + h62.py eas-pin MANIFEST deviation D1, in place: the agent-toolkit layer manifest's + branch revs become the commits its comments name; prints the diff + h62.py eas-edit ENV_DIR the documented catalog edit and deviation D2, in place; prints + the diffs (nothing when the env already reads so) + h62.py uids < PODS_JSON every pod's uid, comma-separated + h62.py pick-pod OLD_UIDS < PODS_JSON " " of a Ready pod not in OLD_UIDS; exit 1 if none + h62.py model-mount CONTAINER < POD_JSON + where CONTAINER mounts the model-store PVC + h62.py verify-script SIBLINGS_JSON a self-contained script that checks a weights directory + against the Hub's file list (run in B5's pod) + h62.py trigger EAS_TAGS EAT_TAGS the re-run trigger, from two `git ls-remote --tags` outputs + h62.py pairs-left PAIRS_DIR B24 pairs still to run + h62.py vmtax H50_ANALYSIS H62_ANALYSIS + A1 on the host (H50) and in its guest (H62), described +""" +import difflib +import json +import os +import re +import sys + +PINS = { + # B5: EAS's release candidate and the component table of its release notes (read 2026-10-29). + "eas_repo": "https://github.com/intel/enterprise-ai-solutions.git", + "eas_tag": "v1.2.0-rc.1", + "eas_sha": "5c27fb39a95eecec41aa4e29a1188984e9c726f3", + "ei_branch": "release-1.1", + "ei_sha": "170eb3abb1956de0787727f21e356aef2b13ff24", + "eat_branch": "release-2.0.0", + "eat_sha": "036b66234801e9bd950afd90830821859780d668", + "catalog_name": "gemma-4-26b", + "stock_model_id": "google/gemma-4-26B-A4B-it", + # Deviation D2: the weights both arms serve (crescent amendment M1). + "qat_repo": "google/gemma-4-26B-A4B-it-qat-q4_0-unquantized", + "qat_rev": "f1e06dc520982d9b9edd76859fdb7ab209449949", + # The image enterprise-inference's catalog names for gemma-4-26b (by tag; the digest it served + # on 2026-10-29, the same image as crescent amendment E1's B1). Recorded, not forced. + "vllm_cpu_image": "vllm/vllm-openai-cpu:v0.28.0@sha256:197354e6475e638eb5ccca09beb078488046a82de0e204e7429bf1a804df9647", + # The guests (deviation D3): Ubuntu 24.04 (an OS EAS's configure supports), QEMU in a container. + "cloud_image_url": "https://cloud-images.ubuntu.com/releases/noble/release-20261017/ubuntu-24.04-server-cloudimg-amd64.img", + "cloud_image_sha256": "04e8f2461a16a44e78a370284471f29616cdd9baf1893df25807b71f985b6c94", + "cloud_image_bytes": 621284352, + "qemu_base": "ubuntu:26.04@sha256:6a06111c3b0a8600feadb270aec771dc239bf407b25625f93f44ab00e11f4d3c", + "qemu_pkg": "1:10.2.1+ds-1ubuntu3.3", + "genisoimage_pkg": "9:1.1.11-5", + # A1's guest: Docker CE from Docker's apt repository, and H50's node image. + "docker_ce": "5:29.9.2-1~ubuntu.24.04~noble", + "containerd": "2.4.2-3~ubuntu.24.04~noble", + "docker_gpg_fpr": "9DC858229FC7DD38854AE2D88D81803C0EBFCD88", + "python3_yaml": "6.0.1-2build2", # bench/apb/stage.py in VM a + "jq": "1.7.1-3ubuntu0.24.04.2", # integrations/eat/eat-check.sh in VM b (the smoke) + "node_image": "cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c", +} +QEMU_IMAGE = "h62-qemu:" + re.sub(r"[^A-Za-z0-9_.-]", "-", PINS["qemu_pkg"]) + +VCPUS = 96 +SMP = "96,sockets=1,cores=48,threads=2" # crescent's topology: 1 socket, 48 cores, 2 threads +HOST_RESERVE_GIB = 48 # MemAvailable left to the host's own tenants +ROOT_MAX_PCT = 90 +REACH = {"huggingface.co": "https://huggingface.co/api/models/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized", + "registry-1.docker.io": "https://registry-1.docker.io/v2/", + "registry.k8s.io": "https://registry.k8s.io/v2/"} +VMS = { + "a": {"name": "h62-vm-a", "host": "h62-a", "arm": "A1", "mem_gib": 96, "disk_gib": 200, + "ssh_port": 18822, "serve_port": 18831, "share": True}, + "b": {"name": "h62-vm-b", "host": "h62-b", "arm": "B5", "mem_gib": 192, "disk_gib": 400, + "ssh_port": 18823, "serve_port": 18832, "share": False}, +} +ARMS = { + # k3d-gpu's agent OVMS (agent manifest: Deployment ovms, label app=ovms, Service ovms:8000). + "A1": {"vm": "a", "ns": "k3d-gpu-agent", "selector": "app=ovms", "svc": "svc/ovms", "svc_port": 8000, + "container": "ovms", "extra": ""}, + # EAS's gemma-4-26b: a KServe LLMInferenceService; KServe v0.18.0 labels its workload pods so + # (GetWorkloadLabelSelector) and serves them behind -kserve-workload-svc:8000. + "B5": {"vm": "b", "ns": "llm-inference", + "selector": "app.kubernetes.io/part-of=llminferenceservice,app.kubernetes.io/name=gemma-4-26b," + "kserve.io/component=workload", + "svc": "svc/gemma-4-26b-kserve-workload-svc", "svc_port": 8000, "container": "main", + "extra": "llminferenceservice/gemma-4-26b"}, +} +TASKS = 24 # B24's task ids, one pair each + + +def arm_fields(arm): + """What bench/h62/arm.sh needs, in one call (it runs on every idle-arm read): VM, guest host, + serve port, namespace, selector, Service, Service port, container, then the extra object.""" + a = ARMS[arm] + v = VMS[a["vm"]] + return [a["vm"], v["host"], v["serve_port"], a["ns"], a["selector"], a["svc"], a["svc_port"], a["container"], + a["extra"]] + + +def mem_needed_kib(): + return (sum(v["mem_gib"] for v in VMS.values()) + HOST_RESERVE_GIB) * 1024 * 1024 + + +def host_facts(meminfo="/proc/meminfo", df_text=None, probe=None): + """MemAvailable (KiB), how full / is (df's percentage) and, per REACH host, whether its URL + answered over HTTPS at all (any HTTP status counts: a registry answers 401).""" + import subprocess + with open(meminfo) as f: + mem = {k: v for k, _, v in (ln.partition(":") for ln in f)} + if df_text is None: + df_text = subprocess.run(["df", "-P", "/"], capture_output=True, text=True).stdout + pct = int(df_text.splitlines()[-1].split()[4].rstrip("%")) + if probe is None: + def probe(url): + return subprocess.run(["curl", "-sS", "-o", "/dev/null", "--max-time", "20", url], + capture_output=True).returncode == 0 + return {"mem_available_kib": int(mem["MemAvailable"].split()[0]), "root_used_pct": pct, + "reach": {h: bool(probe(u)) for h, u in REACH.items()}} + + +def ir_verify(d): + """(lines, bad) of A1's IR directory against its manifest.json: every listed file's sha256 + and size, and the files the manifest does not list.""" + import hashlib + files = _load(os.path.join(d, "manifest.json"))["files"] + out, bad = [], 0 + for n, want in sorted(files.items()): + p = os.path.join(d, n) + h = hashlib.sha256() + with open(p, "rb") as f: + for chunk in iter(lambda: f.read(1 << 24), b""): + h.update(chunk) + ok = h.hexdigest() == want["sha256"] and os.path.getsize(p) == want["bytes"] + bad += not ok + out.append(f"{'ok ' if ok else 'BAD'} {n} {h.hexdigest()} {want['bytes']}") + extra = sorted(set(os.listdir(d)) - set(files) - {"manifest.json"}) + out.append(f"dir {d}; {len(files)} files, {bad} bad; not in the manifest: {extra}") + return out, bad + + +def ssh_config(state): + out = [] + for v in VMS.values(): + out += [f"Host {v['host']}", " HostName 127.0.0.1", f" Port {v['ssh_port']}", " User ubuntu", + f" IdentityFile {state}/id_ed25519", " IdentitiesOnly yes", f" UserKnownHostsFile {state}/known_hosts", + " StrictHostKeyChecking accept-new", " BatchMode yes", " ControlMaster auto", + f" ControlPath {state}/cm-%C", " ControlPersist 600", " ServerAliveInterval 30", ""] + return "\n".join(out) + + +def _key(pubkey): + k = pubkey.strip() + if not re.fullmatch(r"ssh-(ed25519|rsa) [A-Za-z0-9+/=]+( [^\n]*)?", k): + raise ValueError("not one ssh public key line") + return k + + +def user_data(vm, pubkey): + """cloud-init for one guest: the harness's key on the default user, no password login, and + the image's package state kept (deviation D3): no upgrade at boot and no apt timer running + alongside the installs.""" + return "\n".join([ + "#cloud-config", + f"hostname: {VMS[vm]['host']}", + "ssh_pwauth: false", + "ssh_authorized_keys:", + f" - {_key(pubkey)}", + "package_update: false", + "package_upgrade: false", + "runcmd:", + " - [systemctl, disable, --now, unattended-upgrades.service, apt-daily.timer, apt-daily-upgrade.timer]", + ""]) + + +def meta_data(vm): + h = VMS[vm]["host"] + return f"instance-id: {h}\nlocal-hostname: {h}\n" + + +def qemu_argv(vm, state, image, share=None, uid=None, gid=None, kvm_gid=None): + """The guest's QEMU, in the pinned container: KVM, -cpu host (AMX passes through), the host's + topology, user-mode networking with the ssh and serve ports on 127.0.0.1 only, a qcow2 overlay + on the cloud image, and for VM a A1's IR shared read-only.""" + v = VMS[vm] + if v["share"] and not share: + raise ValueError(f"VM {vm} needs A1's IR directory to share") + if share and not v["share"]: + raise ValueError(f"VM {vm} shares nothing") + if None in (uid, gid, kvm_gid): + raise ValueError("uid, gid and the kvm group are required") + argv = ["docker", "run", "-d", "--name", v["name"], "--network", "host", "--device", "/dev/kvm", + "--user", f"{uid}:{gid}", "--group-add", str(kvm_gid), "--label", "io.k3d-gpu.h62=vm", + "-v", f"{state}:/h62"] + if share: + argv += ["-v", f"{share}:/share:ro"] + fwd = f"hostfwd=tcp:127.0.0.1:{v['ssh_port']}-:22,hostfwd=tcp:127.0.0.1:{v['serve_port']}-:{v['serve_port']}" + argv += [image, "qemu-system-x86_64", "-name", v["name"], "-machine", "q35,accel=kvm", "-cpu", "host", + "-smp", SMP, "-m", f"{v['mem_gib']}G", + "-drive", f"file=/h62/{vm}/disk.qcow2,if=virtio,format=qcow2", + "-drive", f"file=/h62/{vm}/seed.iso,if=virtio,format=raw,readonly=on", + "-netdev", f"user,id=n0,{fwd}", "-device", "virtio-net-pci,netdev=n0"] + if share: + argv += ["-virtfs", "local,path=/share,mount_tag=h62share,security_model=none,readonly=on"] + return argv + ["-display", "none", "-serial", "stdio", "-monitor", "none"] + + +def gates(phase, f): + """[(ok, message)] of one gate phase, from its facts.""" + out = [] + if phase == "host": + need = mem_needed_kib() + have = f.get("mem_available_kib") + out.append((have is not None and have >= need, + f"MemAvailable {have} KiB >= {need} KiB (both guests + {HOST_RESERVE_GIB} GiB)")) + pct = f.get("root_used_pct") + out.append((pct is not None and pct <= ROOT_MAX_PCT, f"/ used {pct}% <= {ROOT_MAX_PCT}%")) + reach = f.get("reach") or {} + for h in REACH: + out.append((reach.get(h) is True, f"{h} reachable")) + elif phase == "guest": + for vm in VMS: + amx = (f.get("amx_tile") or {}).get(vm) + out.append((amx == VCPUS, f"VM {vm}: amx_tile on {amx} of {VCPUS} vCPUs")) + n = (f.get("nproc") or {}).get(vm) + out.append((n == VCPUS, f"VM {vm}: nproc {n} == {VCPUS}")) + elif phase == "nfd": + out.append((f.get("amxbf16") == "true", f"VM b node label cpu-cpuid.AMXBF16={f.get('amxbf16')}")) + elif phase == "weights": + ok = f.get("bad") == 0 and f.get("missing") == 0 and (f.get("checked") or 0) > 0 + out.append((ok, f"B5's weights: {f.get('checked')} LFS files checked, {f.get('bad')} differ, " + f"{f.get('missing')} missing, against {PINS['qat_repo']}@{PINS['qat_rev']}")) + else: + raise ValueError(f"no gate phase {phase}") + return out + + +def _diff(old, new, name): + return "".join(difflib.unified_diff(old.splitlines(True), new.splitlines(True), name, name)) + + +def eas_pin(text): + """Deviation D1: in EAS's agent-toolkit layer manifest, each layer's branch rev becomes the + commit Intel's own comment on that line names (the rc.1 release notes' component table). + Refuses anything else: another branch, a comment naming another commit, a missing layer.""" + want = {"inference": (PINS["ei_branch"], PINS["ei_sha"]), "agent-toolkit": (PINS["eat_branch"], PINS["eat_sha"])} + lines = text.splitlines(True) + starts = [i for i, ln in enumerate(lines) if re.match(r'\s*- layer: "', ln)] + seen = set() + for k, i in enumerate(starts): + layer = re.match(r'\s*- layer: "([^"]+)"', lines[i]).group(1) + if layer not in want: + continue + seen.add(layer) + branch, sha = want[layer] + end = starts[k + 1] if k + 1 < len(starts) else len(lines) + revs = [j for j in range(i, end) if re.match(r"\s*rev:", lines[j])] + if len(revs) != 1: + raise ValueError(f"layer {layer}: {len(revs)} rev lines, want 1") + j = revs[0] + indent = re.match(r"(\s*)", lines[j]).group(1) + pinned = f'{indent}rev: "{sha}" # H62 D1, was "{branch}"\n' + if lines[j] == pinned: + continue + m = re.fullmatch(r'\s*rev: "([^"]+)" # pin ([0-9a-f]{40}) .*\n?', lines[j]) + if not m or m.group(1) != branch or m.group(2) != sha: + raise ValueError(f"layer {layer}: rev line is not {branch!r} pinned to {sha}: {lines[j].strip()}") + lines[j] = pinned + if seen != set(want): + raise ValueError(f"layers missing from the manifest: {sorted(set(want) - seen)}") + return "".join(lines) + + +def set_autodeploy(text, name): + """The documented edit (single-node-deployment.md step 3): the one live + llm_services_autodeploy_models line names one catalog value.""" + pat = re.compile(r"^llm_services_autodeploy_models:.*$", re.M) + if len(pat.findall(text)) != 1: + raise ValueError("want exactly one live llm_services_autodeploy_models line") + return pat.sub(f'llm_services_autodeploy_models: ["{name}"]', text) + + +def set_model_id(text, name, old_id, new_id): + """Deviation D2: the model_id of catalog entry `name` (and only that entry) moves from the + stock id to new_id. Refuses when the entry or its stock id is not there.""" + lines = text.splitlines(True) + heads = [i for i, ln in enumerate(lines) if re.match(r"\s*- name:\s*\S", ln)] + for k, i in enumerate(heads): + if re.match(r"\s*- name:\s*(\S+)\s*$", lines[i]).group(1) != name: + continue + end = heads[k + 1] if k + 1 < len(heads) else len(lines) + ids = [j for j in range(i, end) if re.match(r"\s*model_id:", lines[j])] + if len(ids) != 1: + raise ValueError(f"{name}: {len(ids)} model_id lines") + j = ids[0] + indent, value = re.match(r"(\s*)model_id:\s*(\S+)", lines[j]).groups() + if value == new_id: + return text + if value != old_id: + raise ValueError(f"{name}: model_id is {value}, not the stock {old_id}") + lines[j] = f"{indent}model_id: {new_id}\n" + return "".join(lines) + raise ValueError(f"no catalog entry {name}") + + +def eas_edit(env_dir): + """Both edits on an env directory; returns their diffs ("" when nothing changed).""" + out = [] + for name, fn in (("config.agent-toolkit.yaml", lambda t: set_autodeploy(t, PINS["catalog_name"])), + ("models.yaml", lambda t: set_model_id(t, PINS["catalog_name"], PINS["stock_model_id"], + PINS["qat_repo"]))): + p = os.path.join(env_dir, name) + with open(p) as f: + old = f.read() + new = fn(old) + if new != old: + with open(p, "w") as f: + f.write(new) + out.append(_diff(old, new, name)) + return "".join(out) + + +def _ready(item): + st = item.get("status") or {} + return (st.get("phase") == "Running" and not (item.get("metadata") or {}).get("deletionTimestamp") + and any(c.get("type") == "Ready" and c.get("status") == "True" for c in st.get("conditions") or [])) + + +def pick_pod(doc, old): + """(name, uid) of the newest Running, Ready, not-deleting pod whose uid is not in old.""" + new = [i for i in doc.get("items", []) if i["metadata"]["uid"] not in old and _ready(i)] + if not new: + return None + m = max(new, key=lambda i: i["metadata"].get("creationTimestamp", ""))["metadata"] + return m["name"], m["uid"] + + +def model_mount(pod, container, claim="model-store"): + """Where `container` mounts the PVC `claim` (EAS's model store).""" + vols = {v["name"] for v in pod["spec"].get("volumes", []) + if (v.get("persistentVolumeClaim") or {}).get("claimName") == claim} + for c in pod["spec"].get("containers", []): + if c["name"] == container: + for m in c.get("volumeMounts", []): + if m["name"] in vols: + return m["mountPath"] + raise ValueError(f"container {container} mounts no volume of PVC {claim}") + + +VERIFY = r'''#!/usr/bin/env python3 +# Generated by bench/h62/h62.py verify-script: DIR against the Hub's file list of +# @REPO@@@REV@ (H62, B5's weights). Every LFS file's sha256, every other listed file's size, and +# the commit `hf download --local-dir` recorded per file. Last line: a JSON summary. +import glob, hashlib, json, os, sys +SIB = json.loads(@SIB@) +d = sys.argv[1] +bad = missing = checked = 0 +for s in SIB: + p = os.path.join(d, s["rfilename"]) + lfs = s.get("lfs") + if not os.path.exists(p): + print(("MISSING " if lfs else "absent ") + s["rfilename"]) + missing += bool(lfs) + continue + if lfs: + h = hashlib.sha256() + with open(p, "rb") as f: + for chunk in iter(lambda: f.read(1 << 24), b""): + h.update(chunk) + ok = h.hexdigest() == lfs["sha256"] + checked += 1 + print(("ok " if ok else "BAD ") + h.hexdigest() + " " + s["rfilename"]) + else: + ok = os.path.getsize(p) == s.get("size", os.path.getsize(p)) + print(("ok " if ok else "BAD ") + "size " + str(os.path.getsize(p)) + " " + s["rfilename"]) + bad += not ok +revs = set() +for m in glob.glob(os.path.join(d, ".cache", "huggingface", "download", "**", "*.metadata"), recursive=True): + with open(m) as f: + revs.add(f.readline().strip()) +print(json.dumps({"want": "@REV@", "revisions": sorted(revs), "checked": checked, "bad": bad, "missing": missing})) +sys.exit(1 if bad or missing else 0) +''' + + +def verify_script(siblings): + """A stdlib script that checks a weights directory against `siblings` (the Hub API's + /api/models//revision/?blobs=true answer); it needs no network.""" + sib = [{k: s[k] for k in ("rfilename", "size", "lfs") if k in s} for s in siblings["siblings"]] + return (VERIFY.replace("@SIB@", repr(json.dumps(sib))).replace("@REPO@", PINS["qat_repo"]) + .replace("@REV@", PINS["qat_rev"])) + + +def _tags(ls_remote): + return {m.group(1) for m in re.finditer(r"refs/tags/([^\s^]+)(?:\^\{\})?\s*$", ls_remote, re.M)} + + +def trigger(eas_ls_remote, eat_ls_remote): + """The re-run trigger: an EAS v1.2.x release tag (not a release candidate), or an EAT 2.x + release tag (EAT tags its releases release-X.Y.Z; a vX.Y.Z tag counts too).""" + eas = sorted(t for t in _tags(eas_ls_remote) if re.fullmatch(r"v1\.2\.\d+", t)) + eat = sorted(t for t in _tags(eat_ls_remote) if re.fullmatch(r"(release-|v)?2\.\d+\.\d+", t)) + return {"eas": eas, "eat": eat} + + +def pairs_left(pairs_dir, total=TASKS): + """B24 pairs not yet recorded (complete or void): pairs resume across holds (pairs.next_pair).""" + sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "eatflow")) + import pairs # noqa: E402 (bench/eatflow/pairs.py) + return max(0, total - pairs.next_pair(pairs_dir)) + + +def vmtax(h50, h62): + """A1 on the host (H50's analysis) against A1 in its guest (H62's): described, never gated.""" + def arm(a): + s = a["within_budget_success"]["A1"] + return {"success": f"{s['k']}/{s['n']}", "attempt_median_s": a["attempt_median_s"]["A1"]} + host, guest = arm(h50), arm(h62) + r = None + if host["attempt_median_s"] and guest["attempt_median_s"]: + r = round(guest["attempt_median_s"] / host["attempt_median_s"], 3) + return {"a1_host": host, "a1_guest": guest, "attempt_median_guest_over_host": r, "gated": False} + + +def _load(path): + with open(path) as f: + return json.load(f) + + +def main(argv): + if not argv: + print(__doc__) + return 2 + cmd, args = argv[0], argv[1:] + if cmd == "pin": + if args[0] not in PINS: + print(f"h62.py: no pin {args[0]}", file=sys.stderr) + return 2 + print(PINS[args[0]]) + elif cmd == "qemu-image": + print(QEMU_IMAGE) + elif cmd == "vm": + if args[0] not in VMS or args[1] not in VMS[args[0]]: + print(f"h62.py: no VM field {args}", file=sys.stderr) + return 2 + print(VMS[args[0]][args[1]]) + elif cmd == "arm-fields": + print(*arm_fields(args[0])) + elif cmd == "arm": + if args[0] not in ARMS or args[1] not in ARMS[args[0]]: + print(f"h62.py: no arm field {args}", file=sys.stderr) + return 2 + print(ARMS[args[0]][args[1]]) + elif cmd == "ssh-config": + sys.stdout.write(ssh_config(args[0])) + elif cmd == "user-data": + with open(args[1]) as f: + sys.stdout.write(user_data(args[0], f.read())) + elif cmd == "meta-data": + sys.stdout.write(meta_data(args[0])) + elif cmd == "qemu-argv": + vm, state, image, uid, gid, kvm = args[:6] + print("\n".join(qemu_argv(vm, state, image, share=args[6] if len(args) > 6 else None, + uid=int(uid), gid=int(gid), kvm_gid=int(kvm)))) + elif cmd == "host-facts": + print(json.dumps(host_facts())) + elif cmd == "ir-verify": + lines, bad = ir_verify(args[0]) + print("\n".join(lines)) + return 1 if bad else 0 + elif cmd == "gates": + res = gates(args[0], _load(args[1])) + for ok, msg in res: + print(("PASS " if ok else "FAIL ") + msg) + return 0 if all(ok for ok, _ in res) else 1 + elif cmd == "eas-pin": + with open(args[0]) as f: + old = f.read() + new = eas_pin(old) + if new != old: + with open(args[0], "w") as f: + f.write(new) + sys.stdout.write(_diff(old, new, os.path.basename(args[0]))) + elif cmd == "eas-edit": + sys.stdout.write(eas_edit(args[0])) + elif cmd == "uids": + print(",".join(i["metadata"]["uid"] for i in json.load(sys.stdin).get("items", []))) + elif cmd == "pick-pod": + p = pick_pod(json.load(sys.stdin), {u for u in (args[0] if args else "").split(",") if u}) + if not p: + return 1 + print(*p) + elif cmd == "model-mount": + print(model_mount(json.load(sys.stdin), args[0])) + elif cmd == "verify-script": + sys.stdout.write(verify_script(_load(args[0]))) + elif cmd == "trigger": + with open(args[0]) as f, open(args[1]) as g: + t = trigger(f.read(), g.read()) + fired = [f"{k} {v}" for k in ("eas", "eat") for v in t[k]] + print("fired: " + (", ".join(fired) if fired else "none")) + elif cmd == "pairs-left": + print(pairs_left(args[0])) + elif cmd == "vmtax": + print(json.dumps(vmtax(_load(args[0]), _load(args[1])), indent=1)) + else: + print(__doc__) + return 2 + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main(sys.argv[1:])) + except (ValueError, KeyError, IndexError, OSError) as e: + print(f"h62.py: {e}", file=sys.stderr) + sys.exit(2) diff --git a/bench/h62/pairs-hold.sh b/bench/h62/pairs-hold.sh new file mode 100755 index 00000000..1bd0d252 --- /dev/null +++ b/bench/h62/pairs-hold.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# pairs-hold.sh N [smoke]: one H62 hold on crescent (EXPERIMENTS.md "H62 pre-registration") under +# bench/eatflow/crescent/hold.sh: the lock, the hold-start rule, the 97% disk guard, the serving +# sampler and the cleanup, LIVE_CAP 2370 inside a 2400 s timeout. live.sh runs H50's hold with both +# servers in their KVM guests (arm-A1, arm-B5): each hold restarts both serving pods first. +# Records: ~/k3dgpu-bench/results/H62/hold, pairs in ~/k3dgpu-bench/results/H62/pairs, the +# hold's own records in ~/k3dgpu-bench/runs/H62-hold. +# smoke: the wiring smoke instead (H62-smoke, the pair-start guard past the hold so no pair starts, +# records under ~/k3dgpu-bench/smoke/H62); it is not one of the 8 holds. +set -uo pipefail +N=${1:?usage: pairs-hold.sh N [smoke]} +B=${HOME}/k3dgpu-bench +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +case ${2:-hold} in + hold) name=H62-hold${N} R=${B}/results/H62 guard=600 ;; + smoke) name=H62-smoke R=${B}/smoke/H62 guard=99999 ;; + *) echo "usage: pairs-hold.sh N [smoke]"; exit 2 ;; +esac +mkdir -p "${R}/pairs" +# The guests are this campaign's arms: the serving covariate records them as vm_cpu (serving.py). +export SERVING_VMS=h62-vm-a,h62-vm-b +exec "${HERE}/../eatflow/crescent/hold.sh" "${name}" 2400 env \ + LIVE_MODE=pairs LIVE_CAP=2370 LIVE_HOLD="${N}" LIVE_PAIRS_DIR="${R}/pairs" LIVE_PAIRS_GUARD="${guard}" \ + LIVE_MODEL_SET=gemma4 LIVE_MODELS="${B}/models" LIVE_BLUEPRINT="${B}/eab" LIVE_PY="${B}/mafvenv/bin/python" \ + LIVE_REMOTE_A="${HERE}/arm-A1" LIVE_REMOTE_B="${HERE}/arm-B5" H62_STATE="${H62_STATE:-${B}/h62}" \ + PATH="${B}/bin:${PATH}" \ + bash "${HERE}/../eatflow/live.sh" H62 "${R}/hold${N}" diff --git a/bench/h62/qemu.Dockerfile b/bench/h62/qemu.Dockerfile new file mode 100644 index 00000000..e65ead46 --- /dev/null +++ b/bench/h62/qemu.Dockerfile @@ -0,0 +1,12 @@ +# The QEMU that runs H62's two KVM guests on crescent (EXPERIMENTS.md "H62 pre-registration", +# deviation D3), in a container so the host gets no package: Ubuntu 26.04's qemu-system-x86 (QEMU +# 10.2, which passes AMX through to a guest from QEMU 7.0 on), qemu-img and genisoimage for the +# cloud-init seed. Every version is a pin of bench/h62/h62.py (PINS); bench/h62/vm.sh builds it. +FROM ubuntu:26.04@sha256:6a06111c3b0a8600feadb270aec771dc239bf407b25625f93f44ab00e11f4d3c +RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + qemu-system-x86=1:10.2.1+ds-1ubuntu3.3 \ + qemu-utils=1:10.2.1+ds-1ubuntu3.3 \ + genisoimage=9:1.1.11-5 \ + && dpkg-query -W > /h62-packages.txt \ + && rm -rf /var/lib/apt/lists/* diff --git a/bench/h62/setup.sh b/bench/h62/setup.sh new file mode 100755 index 00000000..fdb27e40 --- /dev/null +++ b/bench/h62/setup.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# setup.sh: H62's setup phase on crescent (EXPERIMENTS.md "H62 pre-registration", "Setup"), not a +# hold, inside the bench lock. Run from the H62 checkout at the deploy commit. +# 1. host gate: MemAvailable for both guests plus 48 GiB, / at most 90% used, Hugging Face and the +# registries reachable; re-read every 10 min for up to 60 min +# 2. A1's IR against its manifest.json; the Hub's file list of B5's weights (the QAT checkpoint +# at f1e06dc5) and the script that checks them +# 3. both guests (vm.sh build, fetch, up, wait), then the guest gate: amx_tile on all 96 vCPUs +# 4. VM a's A1 (guest-a.sh, 45 min ceiling) and VM b's EAS install (guest-b.sh, 120 min ceiling), +# side by side; their records copied back +# 5. the NFD gate (VM b's node carries cpu-cpuid.AMXBF16=true) and the weight gate (B5's files +# equal the QAT checkpoint's) +# Exit 0: both arms serve. 4: lock not taken. 6: the host gate still failed after 60 min. 7: not +# deliverable (a ceiling passed, or no AMX in a guest). 8: the NFD or weight gate failed (no hold +# until a dated amendment). 1: anything else. Records: ~/k3dgpu-bench/results/H62/setup. +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +P() { python3 "${HERE}/h62.py" "$@"; } +B=${HOME}/k3dgpu-bench +export H62_STATE=${H62_STATE:-${B}/h62} +R=${H62_RESULTS:-${B}/results/H62}/setup +A_CAP=2700 B_CAP=7200 +mkdir -p "${R}" +log() { echo "== setup $* $(date -u +%FT%TZ)" | tee -a "${R}/setup.log"; } +# Every ssh, scp and vm.sh below runs with fd 9 closed: an ssh ControlMaster it starts outlives +# this script, and must not keep the bench lock. +G() { ssh -F "${H62_STATE}/ssh_config" "$(P vm "$1" host)" "${@:2}" 9>&-; } +put() { scp -q -F "${H62_STATE}/ssh_config" "${@:2}" "$(P vm "$1" host):h62/" 9>&-; } +vm() { bash "${HERE}/vm.sh" "$@" 9>&-; } +exec 9> "${HOME}/k3dgpu-bench.lock" +flock -w 7200 9 || { log "lock not taken in 7200 s"; exit 4; } +log "lock taken" +git -C "${REPO}" rev-parse HEAD > "${R}/commit.txt" +git -C "${REPO}" status --porcelain > "${R}/commit-status.txt" + +# 1. Host gate. +for i in 0 1 2 3 4 5 6; do + P host-facts > "${R}/host-facts.json" + P gates host "${R}/host-facts.json" > "${R}/gate-host.txt" && break + [ "${i}" -lt 6 ] || { log "host gate failed after 60 min"; cat "${R}/gate-host.txt"; exit 6; } + log "host gate not passed: $(grep -c FAIL "${R}/gate-host.txt") item(s); re-read in 10 min" + sleep 600 +done +log "host gate passed" + +# 2. A1's IR and B5's reference list. +IR=$(readlink -f "${H62_IR:-${B}/models/gemma-4-26B-A4B-it-qat-int4-ov}") +P ir-verify "${IR}" > "${R}/ir-verify.txt" || { log "A1's IR does not match its manifest"; exit 1; } +curl -fsSL "https://huggingface.co/api/models/$(P pin qat_repo)/revision/$(P pin qat_rev)?blobs=true" \ + > "${R}/qat-siblings.json" || { log "no Hub file list"; exit 1; } +P verify-script "${R}/qat-siblings.json" > "${R}/verify-weights.py" + +# 3. The guests. +log "guests" +vm build > "${R}/vm-build.log" 2>&1 || { log "QEMU image build failed"; exit 1; } +vm fetch > "${R}/vm-fetch.log" 2>&1 || { log "cloud image failed its check"; exit 1; } +for vm in a b; do + H62_IR=${IR} vm up "${vm}" > "${R}/vm-up-${vm}.log" 2>&1 || { log "VM ${vm} did not start"; exit 1; } +done +for vm in a b; do + vm wait "${vm}" 900 >> "${R}/vm-up-${vm}.log" 2>&1 || { log "VM ${vm} did not boot"; exit 1; } +done +fa=$(vm facts a) && fb=$(vm facts b) || { log "no guest facts"; exit 1; } +python3 -c 'import json, sys; a, b = (json.loads(x) for x in sys.argv[1:3]) +print(json.dumps({"amx_tile": {"a": a["amx_tile"], "b": b["amx_tile"]}, "nproc": {"a": a["nproc"], "b": b["nproc"]}, + "a": a, "b": b}))' "${fa}" "${fb}" > "${R}/guest-facts.json" +P gates guest "${R}/guest-facts.json" > "${R}/gate-guest.txt" || { log "guest gate failed: not deliverable"; exit 7; } + +# 4. Each guest's setup, side by side under its ceiling. +for vm in a b; do + G "${vm}" mkdir -p h62/bin + put "${vm}" "${HERE}/h62.py" "${HERE}/guest-${vm}.sh" +done +scp -q -F "${H62_STATE}/ssh_config" "${B}/bin/k3d" "${B}/bin/kubectl" h62-a:h62/bin/ 9>&- +git -C "${REPO}" archive --format=tar HEAD | G a "mkdir -p h62/k3d-gpu && tar -x -C h62/k3d-gpu" +log "guest-a (cap ${A_CAP} s) and guest-b (cap ${B_CAP} s)" +timeout -k 60 "${A_CAP}" ssh -F "${H62_STATE}/ssh_config" h62-a 'bash h62/guest-a.sh' > "${R}/guest-a.log" 2>&1 9>&- & +pa=$! +timeout -k 60 "${B_CAP}" ssh -F "${H62_STATE}/ssh_config" h62-b 'bash h62/guest-b.sh' > "${R}/guest-b.log" 2>&1 9>&- & +pb=$! +wait "${pa}"; xa=$? +wait "${pb}"; xb=$? +for vm in a b; do + scp -q -r -F "${H62_STATE}/ssh_config" "$(P vm "${vm}" host):h62/records" "${R}/guest-${vm}" 2>/dev/null 9>&- || true +done +log "guest-a exit ${xa}, guest-b exit ${xb}" +if [ "${xa}" = 124 ] || [ "${xb}" = 124 ]; then log "a ceiling passed: not deliverable"; exit 7; fi +[ "${xa}" = 0 ] && [ "${xb}" = 0 ] || exit 1 + +# 5. NFD and weight gates on VM b. +amx=$(G b ". ~/h62/env && kubectl get nodes -o jsonpath='{.items[0].metadata.labels.feature\\.node\\.kubernetes\\.io/cpu-cpuid\\.AMXBF16}'") +python3 -c 'import json, sys; print(json.dumps({"amxbf16": sys.argv[1] or None}))' "${amx}" > "${R}/nfd-facts.json" +P gates nfd "${R}/nfd-facts.json" > "${R}/gate-nfd.txt" || { log "NFD gate failed"; exit 8; } +sel=$(P arm B5 selector) +read -r pod _ <<< "$(G b ". ~/h62/env && kubectl -n llm-inference get pods -l ${sel} -o json" | P pick-pod "")" +[ -n "${pod:-}" ] || { log "no Ready B5 pod"; exit 1; } +mnt=$(G b ". ~/h62/env && kubectl -n llm-inference get pod ${pod} -o json" | P model-mount main) || { log "no model mount"; exit 1; } +G b ". ~/h62/env && kubectl -n llm-inference exec -i ${pod} -c main -- python3 - ${mnt}" \ + < "${R}/verify-weights.py" > "${R}/weights-verify.txt" 2>&1 +tail -1 "${R}/weights-verify.txt" > "${R}/weights-facts.json" +P gates weights "${R}/weights-facts.json" > "${R}/gate-weights.txt" || { log "weight gate failed"; exit 8; } +log "setup done: both arms serve" diff --git a/bench/h62/smoke.sh b/bench/h62/smoke.sh new file mode 100755 index 00000000..7b5d3024 --- /dev/null +++ b/bench/h62/smoke.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# smoke.sh: H62's wiring smoke (EXPERIMENTS.md "H62 pre-registration", "Wiring smoke"), after +# setup.sh and before hold 1. It scores nothing, is not one of the 8 holds, and is disclosed in +# the run section. +# 1. B5 through EAS's own stack, in VM b, inside the bench lock: `es_auto_installer.sh validate +# agent-toolkit`, then integrations/eat/eat-check.sh in EAS mode (PR #260's interface) through +# EAS's gateway on gemma-4-26b: a chat, a get_weather tool call, and a sandbox run that +# prints 385 +# 2. one H62 hold with the pair-start guard past its end (pairs-hold.sh 0 smoke): both serving +# pods restart and answer through the harness's LiteLLM, both warm-ups answer, no pair starts +# Exit 0 when both pass; 1 otherwise; 4 when the lock is not taken. Records: ~/k3dgpu-bench/smoke/H62. +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +P() { python3 "${HERE}/h62.py" "$@"; } +B=${HOME}/k3dgpu-bench +export H62_STATE=${H62_STATE:-${B}/h62} +S=${B}/smoke/H62 +MODEL=$(P pin catalog_name) +CHECK=${REPO}/integrations/eat/eat-check.sh +mkdir -p "${S}" +log() { echo "== smoke $* $(date -u +%FT%TZ)" | tee -a "${S}/smoke.log"; } +G() { ssh -F "${H62_STATE}/ssh_config" h62-b "$@" 9>&-; } +grep -qE '^[[:space:]]+eas\)' "${CHECK}" || { log "eat-check.sh at this commit has no EAS mode (PR #260): no smoke"; exit 1; } +exec 9> "${HOME}/k3dgpu-bench.lock" +flock -w 7200 9 || { log "lock not taken in 7200 s"; exit 4; } +log "lock taken; step 1 in VM b" +G "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q jq=$(P pin jq)" > "${S}/jq.log" 2>&1 \ + || { log "jq not installed in VM b"; exit 1; } +scp -q -F "${H62_STATE}/ssh_config" "${CHECK}" h62-b:h62/eat-check.sh 9>&- +G "cd ~/enterprise-ai-solutions && ./es_auto_installer.sh validate agent-toolkit" > "${S}/validate.log" 2>&1 +xv=$? +G ". ~/h62/env && EAT_MODE=eas EAT_MODEL=${MODEL} bash ~/h62/eat-check.sh" > "${S}/eat-check.log" 2>&1 +xc=$? +log "validate exit ${xv}, eat-check exit ${xc}" +exec 9>&- +log "step 2: H62-smoke" +bash "${HERE}/pairs-hold.sh" 0 smoke > "${S}/hold.log" 2>&1 +xh=$? +warm=$(python3 -c 'import json, sys +rows = [json.loads(x) for x in open(sys.argv[1]) if x.strip()] +print(sum(1 for r in rows if not r.get("error")))' "${S}/pairs/warmups.jsonl" 2>/dev/null || echo 0) +pairs=$(grep -c . "${S}/pairs/pairs.jsonl" 2>/dev/null || true) +log "hold exit ${xh}, warm-ups answered ${warm} of 2, pairs ${pairs:-0}" +[ "${xv}" = 0 ] && [ "${xc}" = 0 ] && [ "${xh}" = 0 ] && [ "${warm}" = 2 ] && [ "${pairs:-0}" = 0 ] || exit 1 +log "wiring smoke passed" diff --git a/bench/h62/teardown.sh b/bench/h62/teardown.sh new file mode 100755 index 00000000..3ca482e7 --- /dev/null +++ b/bench/h62/teardown.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# teardown.sh: H62's teardown (EXPERIMENTS.md "H62 pre-registration"), after the last hold or any +# abort. Both guests are powered off and removed (vm.sh down), and their disks, the cloud image and +# the QEMU image are deleted (vm.sh purge); ~/k3dgpu-bench/results/H62 stays. Serving pairs that +# campaign.sh stopped are started again (gpu-serving.sh start) and then checked through Dynamo +# :8100 as crescent amendment E2 item 7 does: a 16-token chat answers with content and that pair's +# SGLang num_requests_total rises, polled for up to 180 s. The result is recorded; it gates nothing. +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +B=${HOME}/k3dgpu-bench +R=${B}/results/H62 +export H62_STATE=${H62_STATE:-${B}/h62} +mkdir -p "${R}" +for vm in a b; do bash "${HERE}/vm.sh" down "${vm}"; done +bash "${HERE}/vm.sh" purge +if [ -s "${R}/serving-stopped.txt" ]; then + while read -r c; do + bash "${REPO}/bench/eatflow/crescent/gpu-serving.sh" start "${c}" >> "${R}/serving.log" 2>&1 + done < "${R}/serving-stopped.txt" + mv "${R}/serving-stopped.txt" "${R}/serving-restored.txt" + n() { curl -s -m 8 "http://127.0.0.1:$1/metrics" | awk '/^sglang:num_requests_total/{s+=$2} END{print s+0}'; } + for port in 30000 30001; do + before=$(n "${port}") end=$(( SECONDS + 180 )) ok=0 c="" + until [ "${SECONDS}" -ge "${end}" ]; do + c=$(curl -s -m 60 http://127.0.0.1:8100/v1/chat/completions -H 'Content-Type: application/json' \ + -d '{"model":"Qwen3.8-27B","messages":[{"role":"user","content":"Say OK"}],"max_tokens":16,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}' \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["choices"][0]["message"]["content"].strip())' 2>/dev/null) + if [ -n "${c}" ] && [ "$(n "${port}")" -gt "${before}" ]; then ok=1; break; fi + sleep 5 + done + echo "$(date -u +%FT%TZ) :${port} pass=${ok} before=${before} after=$(n "${port}") content=${c:-none}" \ + | tee -a "${R}/frontdoor-post.txt" + done +fi diff --git a/bench/h62/test_h62.py b/bench/h62/test_h62.py new file mode 100644 index 00000000..d17a0411 --- /dev/null +++ b/bench/h62/test_h62.py @@ -0,0 +1,738 @@ +#!/usr/bin/env python3 +"""Tests for bench/h62 (EXPERIMENTS.md "H62 pre-registration"): the pins, the two guests' +QEMU command lines and cloud-init, the setup gates, deviation D1 (the layer manifest's +commits) and the documented edits with deviation D2, the pod and cgroup reads behind the +idle-arm check, B5's weight check, the re-run trigger, and the shell scripts against stubs. +No VM, no cluster, no network.""" +import json +import os +import shutil +import stat +import subprocess +import sys +import tempfile +import textwrap +import unittest + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import h62 # noqa: E402 + +REPO = os.path.normpath(os.path.join(HERE, "..", "..")) +GIB_KIB = 1024 * 1024 + + +def read(*path): + with open(os.path.join(*path)) as f: + return f.read() + + +def run(*argv, stdin=None, env=None): + return subprocess.run([sys.executable, os.path.join(HERE, "h62.py"), *argv], input=stdin, + capture_output=True, text=True, env=env) + + +# EAS v1.2.0-rc.1 (5c27fb3) configs/repos/repos.agent-toolkit.yaml, lines 12-28, verbatim. +MANIFEST = textwrap.dedent('''\ + repos: + - layer: "inference" + url: "https://github.com/intel/enterprise-inference" + dest: "enterprise.ai-inference" + rev: "release-1.1" # pin 170eb3abb1956de0787727f21e356aef2b13ff24 (#49) for release + deployment_subdir: "" + config_dir: "" + default_flavour: "" + model_catalog: "model_manager/models.yaml" + + - layer: "agent-toolkit" + url: "https://github.com/intel/enterprise-agent-toolkit" + dest: "enterprise.agent-toolkit" + rev: "release-2.0.0" # pin 036b66234801e9bd950afd90830821859780d668 (#19) for release + deployment_subdir: "" + config_dir: "" + default_flavour: "" + ''') + +# EAT release-2.0.0 (036b662) config.yaml, lines 33-36 (the file init seeds as +# env/local/config.agent-toolkit.yaml), and the same key commented in a doc block. +CONFIG = textwrap.dedent('''\ + # Model(s) to deploy, from the catalog (env/local/models.yaml) + # llm_services_autodeploy_models: ["qwen3-coder-30b", "bge-base-en"] + llm_services_autodeploy_models: ["qwen2-5-coder-14b"] + redis_stack_enabled: true + ''') + +# enterprise-inference release-1.1 (170eb3a) model_manager/models.yaml, lines 355-366 and +# 405-416, trimmed: the gemma-4-26b entry between two others, at the file's indentation. +MODELS = textwrap.indent(textwrap.dedent('''\ + - name: gemma-4-26b-tp2 + model_id: google/gemma-4-26B-A4B-it + category: llm + + - name: gemma-4-26b + model_id: google/gemma-4-26B-A4B-it + category: llm + runtime: vllm + server_version: "0.28.0" + cpu: auto # one NUMA node per TP rank, per node + memory: 128Gi + + - name: qwen3guard + model_id: Qwen/Qwen3Guard-Gen-0.6B + category: llm + '''), " ") + + +class PinsTest(unittest.TestCase): + """Every pin the pre-registration names, in one table the scripts read.""" + + def test_pins_are_the_preregistered_values(self): + p = h62.PINS + self.assertEqual(p["eas_tag"], "v1.2.0-rc.1") + self.assertEqual(p["eas_sha"], "5c27fb39a95eecec41aa4e29a1188984e9c726f3") + self.assertEqual(p["ei_sha"], "170eb3abb1956de0787727f21e356aef2b13ff24") + self.assertEqual(p["eat_sha"], "036b66234801e9bd950afd90830821859780d668") + self.assertEqual(p["cloud_image_sha256"], "04e8f2461a16a44e78a370284471f29616cdd9baf1893df25807b71f985b6c94") + self.assertTrue(p["cloud_image_url"].endswith("/release-20261017/ubuntu-24.04-server-cloudimg-amd64.img")) + self.assertEqual(p["cloud_image_bytes"], 621284352) + self.assertEqual(p["qemu_base"], "ubuntu:26.04@sha256:6a06111c3b0a8600feadb270aec771dc239bf407b25625f93f44ab00e11f4d3c") + self.assertEqual(p["qemu_pkg"], "1:10.2.1+ds-1ubuntu3.3") + self.assertEqual(p["docker_ce"], "5:29.9.2-1~ubuntu.24.04~noble") + self.assertEqual(p["containerd"], "2.4.2-3~ubuntu.24.04~noble") + self.assertEqual(p["docker_gpg_fpr"], "9DC858229FC7DD38854AE2D88D81803C0EBFCD88") + self.assertEqual(p["node_image"], "cryptoandcoffee/k3d-gpu:0.1.61@sha256:" + "8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c") + self.assertEqual(p["qat_repo"], "google/gemma-4-26B-A4B-it-qat-q4_0-unquantized") + self.assertEqual(p["qat_rev"], "f1e06dc520982d9b9edd76859fdb7ab209449949") + self.assertEqual((p["catalog_name"], p["stock_model_id"]), ("gemma-4-26b", "google/gemma-4-26B-A4B-it")) + self.assertEqual(p["vllm_cpu_image"], "vllm/vllm-openai-cpu:v0.28.0@sha256:" + "197354e6475e638eb5ccca09beb078488046a82de0e204e7429bf1a804df9647") + self.assertEqual((p["jq"], p["python3_yaml"]), ("1.7.1-3ubuntu0.24.04.2", "6.0.1-2build2")) + self.assertEqual(h62.QEMU_IMAGE, "h62-qemu:1-10.2.1-ds-1ubuntu3.3") + + def test_cli_prints_one_pin(self): + r = run("pin", "eas_sha") + self.assertEqual((r.returncode, r.stdout), (0, "5c27fb39a95eecec41aa4e29a1188984e9c726f3\n")) + self.assertEqual(run("pin", "nope").returncode, 2) + + def test_the_qemu_image_is_built_from_the_pins(self): + df = read(HERE, "qemu.Dockerfile") + self.assertIn(f"FROM {h62.PINS['qemu_base']}\n", df) + for pkg in ("qemu-system-x86", "qemu-utils"): + self.assertIn(f"{pkg}={h62.PINS['qemu_pkg']}", df) + self.assertIn(f"genisoimage={h62.PINS['genisoimage_pkg']}", df) + + def test_the_cloud_image_is_checked_before_use(self): + vm = read(HERE, "vm.sh") + self.assertIn('P pin cloud_image_sha256', vm) + self.assertIn('sha256sum -c', vm) + + +class VmSpecTest(unittest.TestCase): + def test_two_guests_of_one_shape(self): + a, b = h62.VMS["a"], h62.VMS["b"] + self.assertEqual((a["arm"], a["mem_gib"], b["arm"], b["mem_gib"]), ("A1", 96, "B5", 192)) + self.assertEqual(h62.SMP, "96,sockets=1,cores=48,threads=2") + self.assertEqual(h62.VCPUS, 96) + # The guests carry no k3dgpu prefix: hold.sh's cleanup never removes them, and another + # row's serving covariate counts them as foreign load. + for v in (a, b): + self.assertFalse(v["name"].startswith("k3dgpu"), v["name"]) + self.assertEqual(len({a["ssh_port"], a["serve_port"], b["ssh_port"], b["serve_port"]}), 4) + + def test_memory_gate_is_both_guests_plus_48_gib(self): + self.assertEqual(h62.mem_needed_kib(), (96 + 192 + 48) * GIB_KIB) + + +class QemuArgvTest(unittest.TestCase): + def test_vm_b(self): + argv = h62.qemu_argv("b", "/st", "h62-qemu:x", uid=1000, gid=1000, kvm_gid=992) + self.assertEqual(argv[:4], ["docker", "run", "-d", "--name"]) + self.assertEqual(argv[4], "h62-vm-b") + for pair in (["--network", "host"], ["--device", "/dev/kvm"], ["--user", "1000:1000"], + ["--group-add", "992"], ["-v", "/st:/h62"], ["-cpu", "host"], ["-m", "192G"], + ["-smp", "96,sockets=1,cores=48,threads=2"], ["-machine", "q35,accel=kvm"]): + self.assertTrue(any(argv[i:i + 2] == pair for i in range(len(argv))), pair) + net = argv[argv.index("-netdev") + 1] + self.assertEqual(net, "user,id=n0,hostfwd=tcp:127.0.0.1:18823-:22,hostfwd=tcp:127.0.0.1:18832-:18832") + self.assertNotIn("0.0.0.0", " ".join(argv)) + self.assertNotIn("-virtfs", argv) + self.assertIn("file=/h62/b/disk.qcow2,if=virtio,format=qcow2", argv) + self.assertIn("file=/h62/b/seed.iso,if=virtio,format=raw,readonly=on", argv) + self.assertEqual(argv[argv.index("h62-qemu:x") + 1], "qemu-system-x86_64") + + def test_vm_a_shares_the_ir_read_only(self): + argv = h62.qemu_argv("a", "/st", "img", share="/home/u/ir", uid=1, gid=1, kvm_gid=2) + self.assertIn("/home/u/ir:/share:ro", argv) + self.assertEqual(argv[argv.index("-virtfs") + 1], + "local,path=/share,mount_tag=h62share,security_model=none,readonly=on") + self.assertEqual(argv[argv.index("-m") + 1], "96G") + with self.assertRaises(ValueError): + h62.qemu_argv("b", "/st", "img", share="/x", uid=1, gid=1, kvm_gid=2) + with self.assertRaises(ValueError): + h62.qemu_argv("a", "/st", "img", uid=1, gid=1, kvm_gid=2) # A1's IR is required + + def test_cli_one_argument_per_line(self): + r = run("qemu-argv", "b", "/st", "img", "1000", "1000", "992") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(r.stdout.splitlines(), h62.qemu_argv("b", "/st", "img", uid=1000, gid=1000, kvm_gid=992)) + + +class CloudInitTest(unittest.TestCase): + KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExample h62" + + def test_user_data(self): + u = h62.user_data("b", self.KEY) + self.assertTrue(u.startswith("#cloud-config\n")) + self.assertIn(f" - {self.KEY}\n", u) + self.assertIn("ssh_pwauth: false\n", u) + self.assertIn("package_upgrade: false\n", u) + self.assertIn("hostname: h62-b\n", u) + # D3: the guest's package state stays the image's; no apt run competes with the install. + for unit in ("unattended-upgrades.service", "apt-daily.timer", "apt-daily-upgrade.timer"): + self.assertIn(unit, u) + + def test_meta_data_and_key_checks(self): + self.assertEqual(h62.meta_data("a"), "instance-id: h62-a\nlocal-hostname: h62-a\n") + for bad in ("", "ssh-ed25519 AAAA\nrm -rf /", "not-a-key"): + with self.assertRaises(ValueError): + h62.user_data("a", bad) + + +class HostFactsTest(unittest.TestCase): + """The host gate's facts: MemAvailable, how full / is, and whether the registries answer.""" + + def test_reads_meminfo_df_and_probes(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + mi = os.path.join(tmp, "meminfo") + with open(mi, "w") as f: + f.write("MemTotal: 527700000 kB\nMemAvailable: 374341632 kB\n") + df = "Filesystem 1024-blocks Used Available Capacity Mounted on\n/dev/x 3600000000 1100000000 2400000000 31% /\n" + seen = [] + f = h62.host_facts(mi, df, lambda url: seen.append(url) or "registry.k8s.io" not in url) + self.assertEqual((f["mem_available_kib"], f["root_used_pct"]), (374341632, 31)) + self.assertEqual(f["reach"], {"huggingface.co": True, "registry-1.docker.io": True, "registry.k8s.io": False}) + self.assertEqual(len(seen), 3) + + +class IrVerifyTest(unittest.TestCase): + """A1's IR against its manifest.json (crescent amendment E3 item 4), before VM a sees it.""" + + def test_manifest_check(self): + import hashlib + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + files = {"openvino_model.bin": b"bin" * 10, "openvino_model.xml": b""} + man = {"files": {}} + for n, data in files.items(): + with open(os.path.join(tmp, n), "wb") as f: + f.write(data) + man["files"][n] = {"sha256": hashlib.sha256(data).hexdigest(), "bytes": len(data)} + with open(os.path.join(tmp, "manifest.json"), "w") as f: + json.dump(man, f) + r = run("ir-verify", tmp) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn("2 files, 0 bad", r.stdout) + with open(os.path.join(tmp, "openvino_model.xml"), "ab") as f: + f.write(b" ") + r = run("ir-verify", tmp) + self.assertEqual(r.returncode, 1) + self.assertIn("1 bad", r.stdout) + + +class SshConfigTest(unittest.TestCase): + def test_both_guests(self): + c = h62.ssh_config("/home/u/k3dgpu-bench/h62") + for host, port in (("h62-a", 18822), ("h62-b", 18823)): + blk = c[c.index(f"Host {host}\n"):] + self.assertIn(f" Port {port}\n", blk) + for line in (" HostName 127.0.0.1\n", " User ubuntu\n", " IdentityFile /home/u/k3dgpu-bench/h62/id_ed25519\n", + " UserKnownHostsFile /home/u/k3dgpu-bench/h62/known_hosts\n", " StrictHostKeyChecking accept-new\n", + " ControlMaster auto\n", " ControlPath /home/u/k3dgpu-bench/h62/cm-%C\n", " BatchMode yes\n"): + self.assertEqual(c.count(line), 2, line) + + +class GatesTest(unittest.TestCase): + HOST = {"mem_available_kib": 357 * GIB_KIB, "root_used_pct": 31, + "reach": {"huggingface.co": True, "registry-1.docker.io": True, "registry.k8s.io": True}} + + def failed(self, phase, facts): + return [m for ok, m in h62.gates(phase, facts) if not ok] + + def test_host(self): + self.assertEqual(self.failed("host", self.HOST), []) + edge = dict(self.HOST, mem_available_kib=h62.mem_needed_kib()) + self.assertEqual(self.failed("host", edge), []) + low = dict(self.HOST, mem_available_kib=h62.mem_needed_kib() - 1) + self.assertEqual(len(self.failed("host", low)), 1) + self.assertEqual(self.failed("host", dict(self.HOST, root_used_pct=90)), []) + self.assertEqual(len(self.failed("host", dict(self.HOST, root_used_pct=91))), 1) + unreach = dict(self.HOST, reach={"huggingface.co": False, "registry-1.docker.io": True, "registry.k8s.io": True}) + self.assertIn("huggingface.co", self.failed("host", unreach)[0]) + + def test_guest_amx_on_every_vcpu(self): + good = {"amx_tile": {"a": 96, "b": 96}, "nproc": {"a": 96, "b": 96}} + self.assertEqual(self.failed("guest", good), []) + self.assertEqual(len(self.failed("guest", {"amx_tile": {"a": 96, "b": 0}, "nproc": {"a": 96, "b": 96}})), 1) + self.assertEqual(len(self.failed("guest", {"amx_tile": {"a": 96}, "nproc": {"a": 96}})), 2) # b unread + + def test_nfd_and_weights(self): + self.assertEqual(self.failed("nfd", {"amxbf16": "true"}), []) + self.assertEqual(len(self.failed("nfd", {"amxbf16": None})), 1) + self.assertEqual(self.failed("weights", {"bad": 0, "missing": 0, "checked": 3}), []) + self.assertEqual(len(self.failed("weights", {"bad": 1, "missing": 0, "checked": 3})), 1) + self.assertEqual(len(self.failed("weights", {"bad": 0, "missing": 0, "checked": 0})), 1) + with self.assertRaises(ValueError): + h62.gates("other", {}) + + def test_cli(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + p = os.path.join(tmp, "f.json") + with open(p, "w") as f: + json.dump(self.HOST, f) + r = run("gates", "host", p) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn("PASS", r.stdout) + with open(p, "w") as f: + json.dump(dict(self.HOST, root_used_pct=95), f) + r = run("gates", "host", p) + self.assertEqual(r.returncode, 1) + self.assertIn("FAIL", r.stdout) + + +class EasPinTest(unittest.TestCase): + """Deviation D1: the layer manifest's branch revs become the commits Intel's own comments + and the rc.1 release notes name; nothing else in the file changes.""" + + def test_pins_both_layers_to_the_commented_commits(self): + out = h62.eas_pin(MANIFEST) + self.assertIn('rev: "170eb3abb1956de0787727f21e356aef2b13ff24" # H62 D1, was "release-1.1"', out) + self.assertIn('rev: "036b66234801e9bd950afd90830821859780d668" # H62 D1, was "release-2.0.0"', out) + diff = [(a, b) for a, b in zip(MANIFEST.splitlines(), out.splitlines()) if a != b] + self.assertEqual(len(diff), 2) + self.assertEqual(len(out.splitlines()), len(MANIFEST.splitlines())) + self.assertEqual(h62.eas_pin(out), out) # a second run changes nothing + + def test_refuses_what_it_was_not_written_for(self): + for bad in (MANIFEST.replace("pin 170eb3ab", "pin 0000000b"), # Intel's comment names another commit + MANIFEST.replace('rev: "release-1.1"', 'rev: "main"'), # another branch + MANIFEST.replace('- layer: "inference"', '- layer: "other"'), + MANIFEST + ' rev: "x"\n'): # a second rev in a block + with self.assertRaises(ValueError): + h62.eas_pin(bad) + + +class EasEditTest(unittest.TestCase): + """The one documented edit (the catalog value) and deviation D2 (gemma-4-26b's weights).""" + + def test_autodeploy_gemma_only_on_the_live_line(self): + out = h62.set_autodeploy(CONFIG, "gemma-4-26b") + self.assertIn('\nllm_services_autodeploy_models: ["gemma-4-26b"]\n', out) + self.assertIn('# llm_services_autodeploy_models: ["qwen3-coder-30b", "bge-base-en"]\n', out) + self.assertEqual(h62.set_autodeploy(out, "gemma-4-26b"), out) + with self.assertRaises(ValueError): + h62.set_autodeploy(CONFIG.replace("llm_services_autodeploy_models: [\"qwen", "#x: [\"qwen"), "gemma-4-26b") + + def test_d2_moves_only_gemma_4_26b_to_the_qat_weights(self): + out = h62.set_model_id(MODELS, "gemma-4-26b", "google/gemma-4-26B-A4B-it", + "google/gemma-4-26B-A4B-it-qat-q4_0-unquantized") + blocks = out.split("\n\n") + self.assertIn("model_id: google/gemma-4-26B-A4B-it\n", blocks[0]) # the tp2 entry keeps its id + self.assertIn("model_id: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized\n", blocks[1]) + self.assertIn("model_id: Qwen/Qwen3Guard-Gen-0.6B\n", blocks[2]) + self.assertEqual(sum(1 for a, b in zip(MODELS.splitlines(), out.splitlines()) if a != b), 1) + with self.assertRaises(ValueError): + h62.set_model_id(MODELS, "nope", "a", "b") + with self.assertRaises(ValueError): # the stock id is not there + h62.set_model_id(MODELS, "gemma-4-26b", "google/other", "b") + + def test_cli_edits_an_env_and_prints_the_diffs(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + for name, text in (("config.agent-toolkit.yaml", CONFIG), ("models.yaml", MODELS), + ("global_config.yaml", 'base_domain_name: "solutions.ai"\n')): + with open(os.path.join(tmp, name), "w") as f: + f.write(text) + r = run("eas-edit", tmp) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('+llm_services_autodeploy_models: ["gemma-4-26b"]', r.stdout) + self.assertIn("+ model_id: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized", r.stdout) + self.assertEqual(read(tmp, "global_config.yaml"), 'base_domain_name: "solutions.ai"\n') # untouched + self.assertEqual(run("eas-edit", tmp).stdout, "") # idempotent + p = os.path.join(tmp, "manifest.yaml") + with open(p, "w") as f: + f.write(MANIFEST) + r = run("eas-pin", p) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('+ rev: "170eb3abb1956de0787727f21e356aef2b13ff24"', r.stdout) + + +def pod(name, uid, ready=True, deleting=False, created="2026-10-30T00:00:00Z", phase="Running"): + meta = {"name": name, "uid": uid, "creationTimestamp": created} + if deleting: + meta["deletionTimestamp"] = "2026-10-30T00:01:00Z" + return {"metadata": meta, "status": {"phase": phase, + "conditions": [{"type": "Ready", "status": "True" if ready else "False"}]}} + + +class PodTest(unittest.TestCase): + def test_pick_a_new_ready_pod(self): + old = pod("p0", "u0", deleting=True) + self.assertIsNone(h62.pick_pod({"items": [old, pod("p1", "u1", ready=False)]}, {"u0"})) + self.assertEqual(h62.pick_pod({"items": [old, pod("p1", "u1")]}, {"u0"}), ("p1", "u1")) + self.assertIsNone(h62.pick_pod({"items": [pod("p0", "u0")]}, {"u0"})) # the old pod is not new + self.assertEqual(h62.pick_pod({"items": [pod("p0", "u0")]}, set()), ("p0", "u0")) + two = {"items": [pod("p1", "u1", created="2026-10-30T00:00:01Z"), pod("p2", "u2", created="2026-10-30T00:00:09Z")]} + self.assertEqual(h62.pick_pod(two, set()), ("p2", "u2")) + self.assertIsNone(h62.pick_pod({"items": [pod("p1", "u1", phase="Pending")]}, set())) + + def test_cli(self): + doc = json.dumps({"items": [pod("p0", "u0", deleting=True), pod("p1", "u1")]}) + self.assertEqual(run("uids", stdin=doc).stdout, "u0,u1\n") + r = run("pick-pod", "u0", stdin=doc) + self.assertEqual((r.returncode, r.stdout), (0, "p1 u1\n")) + self.assertEqual(run("pick-pod", "u0,u1", stdin=doc).returncode, 1) + self.assertEqual(run("pick-pod", "", stdin=doc).stdout, "p1 u1\n") + + def test_model_mount(self): + p = {"spec": {"volumes": [{"name": "home", "emptyDir": {}}, + {"name": "model-cache", "persistentVolumeClaim": {"claimName": "model-store"}}], + "containers": [{"name": "main", "volumeMounts": [ + {"name": "home", "mountPath": "/home"}, + {"name": "model-cache", "mountPath": "/mnt/models", "subPath": "gemma-4-26b"}]}]}} + self.assertEqual(h62.model_mount(p, "main"), "/mnt/models") + with self.assertRaises(ValueError): + h62.model_mount(p, "other") + p["spec"]["volumes"][1]["persistentVolumeClaim"]["claimName"] = "x" + with self.assertRaises(ValueError): + h62.model_mount(p, "main") + + +class VerifyScriptTest(unittest.TestCase): + """B5's weights, read in its pod: every LFS file of the QAT checkpoint at f1e06dc5, and the + commit `hf download --local-dir` recorded for each file.""" + + def setUp(self): + import hashlib + self.tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmp) + self.d = os.path.join(self.tmp, "models") + os.makedirs(os.path.join(self.d, ".cache", "huggingface", "download")) + files = {"model-00001.safetensors": b"w1" * 1000, "model-00002.safetensors": b"w2" * 1000, "config.json": b"{}"} + sib = [] + for n, data in files.items(): + with open(os.path.join(self.d, n), "wb") as f: + f.write(data) + meta = os.path.join(self.d, ".cache", "huggingface", "download", n + ".metadata") + with open(meta, "w") as f: + f.write(h62.PINS["qat_rev"] + "\netag\n1730000000.0\n") + s = {"rfilename": n, "size": len(data)} + if n.endswith(".safetensors"): + s["lfs"] = {"sha256": hashlib.sha256(data).hexdigest(), "size": len(data)} + sib.append(s) + sib.append({"rfilename": "README.md", "size": 10}) # not downloaded: noted + self.sib = {"sha": h62.PINS["qat_rev"], "siblings": sib} + self.script = os.path.join(self.tmp, "verify.py") + with open(self.script, "w") as f: + f.write(h62.verify_script(self.sib)) + + def check(self): + r = subprocess.run([sys.executable, self.script, self.d], capture_output=True, text=True) + return r.returncode, json.loads(r.stdout.splitlines()[-1]) + + def test_matching_weights_pass_and_name_their_commit(self): + rc, s = self.check() + self.assertEqual((rc, s["bad"], s["missing"], s["checked"]), (0, 0, 0, 2)) + self.assertEqual(s["revisions"], [h62.PINS["qat_rev"]]) + self.assertEqual(s["want"], h62.PINS["qat_rev"]) + + def test_a_changed_or_missing_weight_fails(self): + with open(os.path.join(self.d, "model-00002.safetensors"), "ab") as f: + f.write(b"x") + rc, s = self.check() + self.assertEqual((rc, s["bad"]), (1, 1)) + os.unlink(os.path.join(self.d, "model-00001.safetensors")) + rc, s = self.check() + self.assertEqual((rc, s["missing"]), (1, 1)) + + def test_cli_writes_the_script(self): + p = os.path.join(self.tmp, "sib.json") + with open(p, "w") as f: + json.dump(self.sib, f) + r = run("verify-script", p) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(r.stdout, h62.verify_script(self.sib)) + + +class TriggerTest(unittest.TestCase): + """The re-run trigger: EAS v1.2.0 (GA) tagged, or an EAT 2.x release tagged.""" + + EAS = ("1d5686e75f8ec0b1edac206e9ca4db7ddde32250\trefs/tags/v1.2.0-rc.1\n" + "5c27fb39a95eecec41aa4e29a1188984e9c726f3\trefs/tags/v1.2.0-rc.1^{}\n" + "aaaa\trefs/tags/v1.1.0\n") + EAT = "bbbb\trefs/tags/release-1.1.0\ncccc\trefs/tags/release-1.0.3-hotfix\n" + + def test_nothing_fired_today(self): + self.assertEqual(h62.trigger(self.EAS, self.EAT), {"eas": [], "eat": []}) + + def test_ga_or_an_eat_2_release_fires(self): + eas = self.EAS + "dddd\trefs/tags/v1.2.0\ndddd\trefs/tags/v1.2.0^{}\n" + self.assertEqual(h62.trigger(eas, self.EAT)["eas"], ["v1.2.0"]) + self.assertEqual(h62.trigger(self.EAS + "e\trefs/tags/v1.2.1\n", self.EAT)["eas"], ["v1.2.1"]) + self.assertEqual(h62.trigger(self.EAS + "e\trefs/tags/v1.2.0-rc.2\n", self.EAT)["eas"], []) + for tag in ("release-2.0.0", "v2.0.0", "release-2.1.0"): + self.assertEqual(h62.trigger(self.EAS, self.EAT + f"f\trefs/tags/{tag}\n")["eat"], [tag], tag) + + def test_cli(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + for n, t in (("eas", self.EAS), ("eat", self.EAT + "f\trefs/tags/release-2.0.0\n")): + with open(os.path.join(tmp, n), "w") as f: + f.write(t) + r = run("trigger", os.path.join(tmp, "eas"), os.path.join(tmp, "eat")) + self.assertEqual(r.returncode, 0) + self.assertEqual(r.stdout, "fired: eat release-2.0.0\n") + + +class CampaignMathTest(unittest.TestCase): + def test_pairs_left(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + self.assertEqual(h62.pairs_left(tmp), 24) + with open(os.path.join(tmp, "pairs.jsonl"), "w") as f: + f.writelines(json.dumps({"j": j, "status": "complete"}) + "\n" for j in range(10)) + f.write('{"j": 10, "sta') # a line still being written + self.assertEqual(h62.pairs_left(tmp), 14) + self.assertEqual(run("pairs-left", tmp).stdout, "14\n") + + def test_vmtax_is_a_description(self): + h50 = {"within_budget_success": {"A1": {"k": 19, "n": 24, "rate": 19 / 24}}, "attempt_median_s": {"A1": 8.0}} + h62r = {"within_budget_success": {"A1": {"k": 20, "n": 24, "rate": 20 / 24}}, "attempt_median_s": {"A1": 9.2}} + t = h62.vmtax(h50, h62r) + self.assertEqual(t["a1_host"], {"success": "19/24", "attempt_median_s": 8.0}) + self.assertEqual(t["a1_guest"], {"success": "20/24", "attempt_median_s": 9.2}) + self.assertAlmostEqual(t["attempt_median_guest_over_host"], 1.15) + self.assertEqual(t["gated"], False) + + +def stub(path, body): + with open(path, "w") as f: + f.write("#!/usr/bin/env bash\n" + body) + os.chmod(path, os.stat(path).st_mode | stat.S_IXUSR) + + +class ArmScriptTest(unittest.TestCase): + """bench/h62/arm.sh against a stub ssh that answers from canned files: the restart wait, + the cgroup spec the idle-arm check reads, cpustat and the ports.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmp) + self.log = os.path.join(self.tmp, "ssh.log") + # The stub logs each remote command, and answers `get pods` from the next pods-N.json. + stub(os.path.join(self.tmp, "ssh"), textwrap.dedent(f'''\ + cmd="${{@: -1}}" + echo "$*" >> "{self.log}" + case "${{cmd}}" in + *"get pods -l"*) + n=$(cat "{self.tmp}/n" 2>/dev/null || echo 0) + f="{self.tmp}/pods-${{n}}.json"; [ -f "$f" ] || f=$(ls "{self.tmp}"/pods-*.json | sort -V | tail -1) + cat "$f"; echo $(( n + 1 )) > "{self.tmp}/n" ;; + *"find /sys/fs/cgroup"*) echo /sys/fs/cgroup/kubepods.slice/kubepods-podu1.slice ;; + *"cat /x/cpu.stat"*) printf 'usage_usec 77\\n' ;; + *) : ;; + esac + ''')) + self.env = dict(os.environ, H62_SSH=os.path.join(self.tmp, "ssh"), H62_STATE=self.tmp, H62_POLL_S="0.1") + + def arm(self, *args): + return subprocess.run(["bash", os.path.join(HERE, "arm.sh"), *args], capture_output=True, text=True, env=self.env) + + def pods(self, *docs): + for i, d in enumerate(docs): + with open(os.path.join(self.tmp, f"pods-{i}.json"), "w") as f: + json.dump({"items": d}, f) + + def test_one_lookup_per_call(self): + self.assertEqual(run("arm-fields", "A1").stdout, + "a h62-a 18831 k3d-gpu-agent app=ovms svc/ovms 8000 ovms \n") + self.assertEqual(h62.arm_fields("B5")[-1], "llminferenceservice/gemma-4-26b") + self.assertEqual(read(HERE, "arm.sh").count("P arm"), 1) # cpustat runs on every idle read + + def test_ports_and_unknown_arm(self): + self.assertEqual(self.arm("A1", "port").stdout, "18831\n") + self.assertEqual(self.arm("B5", "port").stdout, "18832\n") + self.assertEqual(self.arm("C9", "port").returncode, 2) + + def test_restart_waits_for_a_new_ready_pod(self): + self.pods([pod("p0", "u0")], [pod("p0", "u0", deleting=True)], [pod("p1", "u1", ready=False)], [pod("p1", "u1")]) + r = self.arm("B5", "restart", "30") + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn("ready: p1 u1", r.stdout) + log = read(self.log) + self.assertIn("h62-b", log) + self.assertIn("kubectl -n llm-inference delete pod -l app.kubernetes.io/part-of=llminferenceservice," + "app.kubernetes.io/name=gemma-4-26b,kserve.io/component=workload --wait=false", log) + + def test_restart_not_ready_in_time_exits_3(self): + self.pods([pod("p0", "u0")], [pod("p1", "u1", ready=False)]) + self.assertEqual(self.arm("A1", "restart", "1").returncode, 3) + self.assertEqual(self.arm("A1", "restart", "0").returncode, 3) # no time left at all + self.assertIn("kubectl -n k3d-gpu-agent delete pod -l app=ovms --wait=false", read(self.log)) + + def test_cgroup_spec_and_cpustat(self): + self.pods([pod("p1", "u1")]) + r = self.arm("B5", "cgroup") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(r.stdout, f"exec:{os.path.join(HERE, 'arm.sh')} B5 cpustat " + "/sys/fs/cgroup/kubepods.slice/kubepods-podu1.slice\n") + self.assertIn("-name podu1 -o -name '*podu1.slice'", read(self.log)) + r = self.arm("B5", "cpustat", "/x") + self.assertEqual(r.stdout, "usage_usec 77\n") + + def test_forward_restarts_the_guest_port_forward(self): + r = self.arm("A1", "forward") + self.assertEqual(r.returncode, 0, r.stderr) + log = read(self.log) + self.assertIn("port-forward --address 0.0.0.0 svc/ovms 18831:8000", log) + self.assertIn("pf-A1.pid", log) + self.arm("A1", "unforward") + self.assertIn("kill", read(self.log).splitlines()[-1]) + + +class ScriptsTest(unittest.TestCase): + """The shell scripts: syntax, and what each one is pinned to run.""" + + SCRIPTS = ("arm.sh", "vm.sh", "guest-a.sh", "guest-b.sh", "setup.sh", "smoke.sh", "pairs-hold.sh", + "campaign.sh", "teardown.sh", "upstream.sh") + + def test_syntax(self): + for s in self.SCRIPTS: + r = subprocess.run(["bash", "-n", os.path.join(HERE, s)], capture_output=True, text=True) + self.assertEqual(r.returncode, 0, f"{s}: {r.stderr}") + + def test_shellcheck_when_present(self): + if not shutil.which("shellcheck"): + self.skipTest("no shellcheck") + r = subprocess.run(["shellcheck", "-S", "warning", *(os.path.join(HERE, s) for s in self.SCRIPTS)], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stdout) + + def test_vm_up_runs_the_argv_h62_py_builds(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + with open(os.path.join(tmp, "id_ed25519"), "w") as f: # a key exists: none is made + f.write("private\n") + with open(os.path.join(tmp, "id_ed25519.pub"), "w") as f: + f.write(CloudInitTest.KEY + "\n") + env = dict(os.environ, H62_DRY="1", H62_STATE=tmp, H62_KVM_GID="992") + r = subprocess.run(["bash", os.path.join(HERE, "vm.sh"), "up", "b"], capture_output=True, text=True, env=env) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn("docker run -d --name h62-vm-b --network host --device /dev/kvm", r.stdout) + self.assertIn("-smp 96,sockets=1,cores=48,threads=2 -m 192G", r.stdout) + want = h62.qemu_argv("b", tmp, h62.QEMU_IMAGE, uid=os.getuid(), gid=os.getgid(), kvm_gid=992) + self.assertIn(" ".join(want) + "\n", r.stdout) + self.assertIn("qemu-img create -f qcow2 -F qcow2 -b /h62/base.img /h62/b/disk.qcow2 400G", r.stdout) + self.assertIn("genisoimage -output /h62/b/seed.iso -volid cidata -joliet -rock", r.stdout) + self.assertEqual(read(tmp, "b", "meta-data"), h62.meta_data("b")) + self.assertEqual(read(tmp, "b", "user-data"), h62.user_data("b", CloudInitTest.KEY)) + self.assertEqual(read(tmp, "ssh_config"), h62.ssh_config(tmp)) + + def test_pairs_hold_is_h50s_hold_with_remote_arms(self): + sh = read(HERE, "pairs-hold.sh") + self.assertIn('crescent/hold.sh" "${name}" 2400 env', sh) + self.assertIn("hold) name=H62-hold${N} R=${B}/results/H62 guard=600 ;;", sh) + for item in ("LIVE_MODE=pairs", "LIVE_CAP=2370", 'LIVE_HOLD="${N}"', "LIVE_MODEL_SET=gemma4", + 'bash "${HERE}/../eatflow/live.sh" H62', "SERVING_VMS=h62-vm-a,h62-vm-b"): + self.assertIn(item, sh, item) + for absent in ("LIVE_MOUNT_B", "LIVE_EAT_CHART", "K3D_GPU_IMAGE"): # nothing is deployed on the host + self.assertNotIn(absent, sh, absent) + + def test_live_sh_calls_the_remote_arm_as_one_word(self): + # LIVE_REMOTE_A names a script and its arm: live.sh calls "${RA}" with the arm as one word, + # so arm.sh is called through a two-line wrapper per arm. + for arm in ("A1", "B5"): + w = read(HERE, f"arm-{arm}") + self.assertIn(f'exec "$(dirname "$0")/arm.sh" {arm} "$@"', w) + self.assertTrue(os.access(os.path.join(HERE, f"arm-{arm}"), os.X_OK)) + sh = read(HERE, "pairs-hold.sh") + self.assertIn('LIVE_REMOTE_A="${HERE}/arm-A1"', sh) + self.assertIn('LIVE_REMOTE_B="${HERE}/arm-B5"', sh) + + def test_guest_b_is_intels_documented_path(self): + sh = read(HERE, "guest-b.sh") + steps = ['git clone -q -b "${TAG}" https://github.com/intel/enterprise-ai-solutions.git', + 'h62.py" eas-pin configs/repos/repos.agent-toolkit.yaml', + "./es_auto_installer.sh configure --force", + "./es_auto_installer.sh init agent-toolkit", + 'h62.py" eas-edit env/local', + "./es_auto_installer.sh install agent-toolkit"] + pos = [sh.index(s) for s in steps] + self.assertEqual(pos, sorted(pos), "steps in the documented order") + self.assertIn('[ "$(git rev-parse HEAD)" = "${SHA}" ]', sh) + # prerequisites.md "Node Kernel Settings", verbatim values. + self.assertIn("fs.inotify.max_user_instances = 8192", sh) + self.assertIn("fs.inotify.max_user_watches = 1048576", sh) + # global_config.yaml (base_domain_name and the rest) is only copied into the records. + uses = [ln.strip() for ln in sh.splitlines() if "global_config.yaml" in ln] + self.assertEqual(uses, ["# base_domain_name stays EAS's default: global_config.yaml is not edited", + "for f in global_config.yaml config.agent-toolkit.yaml config.inference.yaml " + "models.yaml inventory/hosts.yaml; do"]) + self.assertIn("HF_TOKEN", sh) # recorded as set or unset, never printed + self.assertNotIn("--only", sh) + self.assertNotIn("existing_kubernetes", sh) + + def test_guest_a_deploys_h50s_a1(self): + sh = read(HERE, "guest-a.sh") + live = read(REPO, "bench", "eatflow", "live.sh") + for item in ("K3D_GPU_AGENT_DEVICE=CPU", "K3D_GPU_AGENT_MODEL=k3dgpu/gemma-4-26B-A4B-it-qat-int4-ov", + "K3D_GPU_AGENT_MODEL_NAME=gemma-4-26b-a4b-it", "K3D_GPU_AGENT_DRAFT=off", + "K3D_GPU_AGENT_TOOL_PARSER=gemma4", "K3D_GPU_AGENT_REASONING_PARSER=''", "K3D_GPU_AUTOQUANT=off"): + self.assertIn(item, sh, item) + # The same model set live.sh gave A1 in H50. + self.assertIn("TGT=k3dgpu/gemma-4-26B-A4B-it-qat-int4-ov TGT_DIR=gemma-4-26B-A4B-it-qat-int4-ov A_TOOL=gemma4 A_DRAFT=off", live) + self.assertIn("ALIAS=google/gemma-4-26B-A4B-it SERVED=gemma-4-26b-a4b-it", live) + self.assertIn('stage.py" ovms', sh) + self.assertIn("K3D_GPU_CLUSTER=k3dgpu-e-h62", sh) + self.assertIn('K3D_GPU_IMAGE="$(P pin node_image)"', sh) + self.assertIn("eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1%", sh) + for pkg in ("docker-ce", "docker-ce-cli"): + self.assertIn(f'{pkg}="$(P pin docker_ce)"', sh) + self.assertIn('containerd.io="$(P pin containerd)"', sh) + self.assertIn("docker_gpg_fpr", sh) + + def test_smoke_gates_b5_through_eas_itself(self): + sh = read(HERE, "smoke.sh") + self.assertIn("./es_auto_installer.sh validate agent-toolkit", sh) + self.assertIn("EAT_MODE=eas", sh) + self.assertIn("MODEL=$(P pin catalog_name)", sh) + self.assertIn("EAT_MODE=eas EAT_MODEL=${MODEL}", sh) + self.assertIn('"${HERE}/pairs-hold.sh" 0 smoke', sh) + hold = read(HERE, "pairs-hold.sh") + self.assertIn("smoke) name=H62-smoke", hold) + self.assertIn("guard=99999", hold) + + def test_campaign_order_and_caps(self): + sh = read(HERE, "campaign.sh") + pos = [sh.index(s) for s in ('"${HERE}/setup.sh"', '"${HERE}/smoke.sh"', '"${HERE}/pairs-hold.sh" "${n}"', + "pairs-report")] + self.assertEqual(pos, sorted(pos)) + self.assertIn("MAX_HOLDS=8", sh) + self.assertIn('h62.py" pairs-left', sh) + self.assertIn("trap teardown EXIT", sh) + self.assertIn("--hyp H62 --arm-a A1 --arm-b B5", sh) + self.assertIn("NOT_DELIVERABLE_MAX=2", sh) + + def test_setup_caps_and_gates(self): + sh = read(HERE, "setup.sh") + self.assertIn('flock -w 7200 9', sh) + self.assertIn("A_CAP=2700", sh) # 45 min + self.assertIn("B_CAP=7200", sh) # 120 min + for phase in ("host", "guest", "nfd", "weights"): + self.assertIn(f"P gates {phase}", sh) + + +if __name__ == "__main__": + unittest.main(verbosity=1) diff --git a/bench/h62/upstream.sh b/bench/h62/upstream.sh new file mode 100755 index 00000000..38fe0d4e --- /dev/null +++ b/bench/h62/upstream.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# upstream.sh: H62's re-run trigger (EXPERIMENTS.md "H62 pre-registration", "Re-run trigger"). +# Reads the tags of EAS and EAT with `git ls-remote` and prints "fired: none", or the release tags +# that fired it: an EAS v1.2.x release (not a release candidate) or an EAT 2.x release. +set -euo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +d=$(mktemp -d) +trap 'rm -rf "${d}"' EXIT +git ls-remote --tags https://github.com/intel/enterprise-ai-solutions.git > "${d}/eas" +git ls-remote --tags https://github.com/intel/enterprise-agent-toolkit.git > "${d}/eat" +python3 "${HERE}/h62.py" trigger "${d}/eas" "${d}/eat" diff --git a/bench/h62/vm.sh b/bench/h62/vm.sh new file mode 100755 index 00000000..f6cb067d --- /dev/null +++ b/bench/h62/vm.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# vm.sh build | fetch | up VM | wait VM SECONDS | facts VM | ssh VM CMD... | down VM | purge +# H62's two KVM guests on crescent (EXPERIMENTS.md "H62 pre-registration"): VM a serves A1, VM b +# serves B5. QEMU runs in the pinned container of qemu.Dockerfile, so the host gets no package. +# Disks, seed images, the ssh key and ssh_config live under H62_STATE (default +# ~/k3dgpu-bench/h62); results do not. Shapes, ports and pins come from h62.py. +# build the QEMU image (h62.py qemu-image); its package list into H62_STATE +# fetch the Ubuntu 24.04 cloud image into H62_STATE/base.img, checked against its sha256 +# up VM the guest's qcow2 overlay, its cloud-init seed and its QEMU container; VM a also +# gets A1's IR read-only (H62_IR, default the IR live.sh's gemma4 set names) +# wait VM SECONDS until the guest answers ssh and cloud-init has finished +# facts VM one JSON line: nproc, amx_tile count, MemTotal and kernel (the guest gate's facts) +# ssh VM CMD... a command in the guest +# down VM power the guest off (at most 120 s), then remove its container +# purge the guests' disks, the cloud image and the QEMU image (refused while a guest runs) +# H62_DRY=1 prints each docker command instead of running it. +set -euo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +P() { python3 "${HERE}/h62.py" "$@"; } +STATE=${H62_STATE:-${HOME}/k3dgpu-bench/h62} +IMAGE=$(P qemu-image) +DRY=${H62_DRY:-0} +run() { if [ "${DRY}" = 1 ]; then echo "$*"; else "$@"; fi; } +in_qemu() { run docker run --rm --user "$(id -u):$(id -g)" -v "${STATE}:/h62" "${IMAGE}" "$@"; } +gssh() { ssh -F "${STATE}/ssh_config" "$(P vm "$1" host)" "${@:2}"; } +exists() { [ "${DRY}" != 1 ] && docker ps -a --format '{{.Names}}' | grep -x "$1" > /dev/null; } +mkdir -p "${STATE}" +chmod 700 "${STATE}" +case ${1:?usage: vm.sh build|fetch|up VM|wait VM SECONDS|facts VM|ssh VM CMD...|down VM|purge} in +build) + run docker build -t "${IMAGE}" -f "${HERE}/qemu.Dockerfile" "${HERE}" + if [ "${DRY}" != 1 ]; then + docker image inspect --format '{{.Id}}' "${IMAGE}" > "${STATE}/qemu-image.txt" + docker run --rm "${IMAGE}" cat /h62-packages.txt > "${STATE}/qemu-packages.txt" + fi ;; +fetch) + if [ ! -f "${STATE}/base.img" ]; then + curl -fsSLo "${STATE}/base.img.part" "$(P pin cloud_image_url)" + mv "${STATE}/base.img.part" "${STATE}/base.img" + fi + echo "$(P pin cloud_image_sha256) ${STATE}/base.img" | sha256sum -c + [ "$(stat -c %s "${STATE}/base.img")" = "$(P pin cloud_image_bytes)" ] || { echo "base.img: wrong size"; exit 1; } + chmod 444 "${STATE}/base.img" ;; +up) + vm=${2:?usage: vm.sh up a|b} + name=$(P vm "${vm}" name) + if exists "${name}"; then echo "${name} exists: vm.sh down ${vm} first"; exit 1; fi + d=${STATE}/${vm} + mkdir -p "${d}" + [ -f "${STATE}/id_ed25519" ] || ssh-keygen -q -t ed25519 -N '' -C h62 -f "${STATE}/id_ed25519" + P ssh-config "${STATE}" > "${STATE}/ssh_config" + # A new guest has a new host key on the same forwarded port. + [ ! -f "${STATE}/known_hosts" ] || ssh-keygen -q -R "[127.0.0.1]:$(P vm "${vm}" ssh_port)" \ + -f "${STATE}/known_hosts" > /dev/null 2>&1 || true + P user-data "${vm}" "${STATE}/id_ed25519.pub" > "${d}/user-data" + P meta-data "${vm}" > "${d}/meta-data" + rm -f "${d}/seed.iso" "${d}/disk.qcow2" + in_qemu genisoimage -output "/h62/${vm}/seed.iso" -volid cidata -joliet -rock "/h62/${vm}/user-data" "/h62/${vm}/meta-data" + in_qemu qemu-img create -f qcow2 -F qcow2 -b /h62/base.img "/h62/${vm}/disk.qcow2" "$(P vm "${vm}" disk_gib)G" + share=() + if [ "$(P vm "${vm}" share)" = True ]; then + share=("$(readlink -f "${H62_IR:-${HOME}/k3dgpu-bench/models/gemma-4-26B-A4B-it-qat-int4-ov}")") + fi + kvm=${H62_KVM_GID:-$(stat -c %g /dev/kvm)} + mapfile -t argv < <(P qemu-argv "${vm}" "${STATE}" "${IMAGE}" "$(id -u)" "$(id -g)" "${kvm}" "${share[@]}") + [ "${#argv[@]}" -gt 10 ] || { echo "no QEMU command line for VM ${vm}"; exit 1; } + run "${argv[@]}" ;; +wait) + vm=${2:?usage: vm.sh wait a|b SECONDS} + end=$(( SECONDS + ${3:?usage: vm.sh wait a|b SECONDS} )) + until gssh "${vm}" true > /dev/null 2>&1; do + [ "${SECONDS}" -lt "${end}" ] || { echo "VM ${vm}: no ssh within ${3} s"; exit 1; } + sleep 5 + done + gssh "${vm}" cloud-init status --wait > /dev/null ;; +facts) + gssh "${2:?usage: vm.sh facts a|b}" python3 - <<'PY' +import json, os, platform +flags = [ln for ln in open("/proc/cpuinfo") if ln.startswith("flags")] +mem = {k: v for k, _, v in (ln.partition(":") for ln in open("/proc/meminfo"))} +print(json.dumps({"nproc": os.cpu_count(), "amx_tile": sum(" amx_tile" in ln for ln in flags), + "mem_total_kib": int(mem["MemTotal"].split()[0]), "kernel": platform.release()})) +PY + ;; +ssh) + gssh "${2:?usage: vm.sh ssh a|b CMD...}" "${@:3}" ;; +down) + vm=${2:?usage: vm.sh down a|b} + name=$(P vm "${vm}" name) + gssh "${vm}" sudo poweroff > /dev/null 2>&1 || true + for _ in $(seq 60); do + [ "$(docker inspect -f '{{.State.Running}}' "${name}" 2>/dev/null)" = true ] || break + sleep 2 + done + run docker rm -f "${name}" > /dev/null 2>&1 || true ;; +purge) + for vm in a b; do + if exists "$(P vm "${vm}" name)"; then echo "VM ${vm} still exists: vm.sh down ${vm} first"; exit 1; fi + done + rm -rf "${STATE:?}/a" "${STATE:?}/b" "${STATE}/base.img" "${STATE}/base.img.part" "${STATE}/known_hosts" + run docker image rm "${IMAGE}" > /dev/null 2>&1 || true ;; +*) + echo "usage: vm.sh build|fetch|up VM|wait VM SECONDS|facts VM|ssh VM CMD...|down VM|purge"; exit 2 ;; +esac diff --git a/docs/intel/technical-report.md b/docs/intel/technical-report.md index 48b1d3e1..4a1dcffa 100644 --- a/docs/intel/technical-report.md +++ b/docs/intel/technical-report.md @@ -1,6 +1,6 @@ # k3d-gpu on Intel hardware: technical report -Status: working report, 2026-10-29. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H52b run (81 rows). +Status: working report, 2026-10-29. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H52b run (82 rows). ## Abstract @@ -11,7 +11,7 @@ This report covers what was measured, and how. Every result but H52, H52b, H25x, - Seventeen hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x is H25 replicated on the Xeon 6741P. H57dm (the Gemma 4 QAT MTP drafter speeds llama.cpp decode on the P-cores) is the newest. - Seventeen are REFUTED. Six of them changed shipped defaults. The newest, H52b, is H52 after `k3d-gpu host-setup --apply` on the Xeon 6741P: DSA was advertised and reached a pod, but IAA got no work queue, the QAT plugin advertised none of the VFs, and `up` advertised two of the four B70s: the serving pairs hold about half the VRAM of every card, and its busy-card filter (half or more) left out the two just over half and kept the two just under it [ledger:H52b]. Before it, H51 ran the graded agents-per-box ramp on the Xeon 6741P: the k3d-gpu CPU stack failed at one agent (turn p95 10.41 s against the 10 s SLO), and so did EAT's stock vLLM CPU arm [ledger:H51]. H50 ran inside EAT's banking flow on the same host: EAT's current stock vLLM CPU arm (release-1.1.0) delivered at least as many tasks within the turn budget as the k3d-gpu CPU stack: 21 of 24 against 19 of 24 [ledger:H50]. - 28 are INCONCLUSIVE, mostly because the shared host throttles or its load was unmatched. H50o (the k3d-gpu CPU stack against EAT's own OVMS chart on the Xeon 6741P) is INCONCLUSIVE because that chart's OVMS 2025.4 crashed on loading Gemma 4 26B-A4B, so no pair ran. H54a and H54d (Gemma 4 26B-A4B QAT on the Xeon 6741P) are INCONCLUSIVE because the stock arm, EAT 1.0.3's vLLM CPU v0.24.0, as this harness rendered EAT's entry (no CPU balloon, requests or limits), stalled in both of H54a's holds, so no arm could be scored against it. EAT 1.1.0's vLLM CPU v0.28.0 was not measured in H54a; H50 ran it in EAT's banking flow. H54b and H54c (one Arc Pro B70 against two, and experts in DRAM with the rest on one B70) are INCONCLUSIVE because their "one card" pods saw all four cards, so no arm was delivered as pre-registered. H57da (k3d-gpu's OVMS pick vs llama.cpp for Gemma 4 QAT on the i9-13900T) did not resolve: the strictly pinned llama.cpp arms tracked host load; H57db did not run (memory). H28x (H28 replicated on the Xeon 6741P) is INCONCLUSIVE: its one pre-registered hold failed in setup and measured nothing; one re-run is registered. H56 (two holds at once in cpuset slices of the Xeon 6741P) is INCONCLUSIVE: in every hold with a sibling, the sibling's banking block ended shortly after the measured block began, so all four were VOID and no interference statistic was scored; crescent rows stay at one hold at a time. -- 19 are PENDING: the i9-13900T thread-pinning follow-ups H57dp and H57de, the Xeon 6741P rows H53, H54q, H60 (Qwen3.8-Flash-Next from host DRAM) and the zero-touch thesis row H58, the cross-host replication H7x, the engine-choice rows H59 (Xeon 6741P), H59d (i9-13900T) and H59w (i7-13700), H55 (SLO-gated admission of an E-core neighbour on the i9-13900T), and the second client host's (wildct, i7-13700) rows H52w, H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. +- 20 are PENDING: the i9-13900T thread-pinning follow-ups H57dp and H57de, the Xeon 6741P rows H53, H54q, H60 (Qwen3.8-Flash-Next from host DRAM), the zero-touch thesis row H58 and H62 (the k3d-gpu CPU stack against Intel's new default, EAS v1.2.0-rc.1 installing EAT 2.0, each in its own KVM guest), the cross-host replication H7x, the engine-choice rows H59 (Xeon 6741P), H59d (i9-13900T) and H59w (i7-13700), H55 (SLO-gated admission of an E-core neighbour on the i9-13900T), and the second client host's (wildct, i7-13700) rows H52w, H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. - Every other Intel family is detected and configured in fixture tests only. None has a measurement. No world-first, superlative or priority claim is made; [docs/provenance.md](../provenance.md) supports none. From 4bf6ae79f70c89cfbea253311067d5fe24edf60e Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Thu, 29 Oct 2026 22:18:46 -0700 Subject: [PATCH 2/2] ledger(H62): name the request path as D4, describe the VM tax on both arms, widen the re-run trigger Review of the H62 pre-registration against Intel's sources, before any H62 step. B5's turns reach EAS's workload Service directly, while Intel's agents reach the model through EAS's LiteLLM gateway with the content-safety guardrail and the hybrid router (EAT release-2.0.0 config.yaml lines 20-28). The pre-registration described the bypass but listed only D1-D3 as deviations from stock. It is now D4, in the row and the section, with its direction: it leans toward B5. One guest shape gives neither arm bare metal, but it does not show the guest costs vLLM CPU and OVMS alike. vmtax now also describes B5 in its guest against H50's B1 on the host (the same vLLM CPU v0.28.0 image and QAT weights under another deployment), never gated, and the risk is named. The re-run trigger fired only on EAS v1.2.x and EAT 2.x tags, so a GA shipped as EAS v1.3.0 or v2.0.0, or an EAT hotfix tag, would not re-run H62. It now fires on any EAS release at v1.2.0 or later and any EAT release at 2.0.0 or later, release candidates excluded. The section also records that EAT's README clones EAS main, which names EI main and EAT branch agentic-toolkit-migration, and that KServe v0.21.0 is newer than EI's v0.18.0. Predictions are unchanged. --- EXPERIMENTS.md | 54 +++++++++++++++++++++++++++++++------------ bench/h62/h62.py | 48 +++++++++++++++++++++++++++----------- bench/h62/test_h62.py | 40 ++++++++++++++++++++++++++++---- 3 files changed, 109 insertions(+), 33 deletions(-) diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 760d7873..3a2d481c 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -68,7 +68,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H50 | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) On EAT's own target CPU (Xeon 6741P, Granite Rapids, AMX), inside EAT's banking flow, the k3d-gpu CPU stack A1 (OVMS int4 ~~with the CPU Fast Draft~~, `agent manifest`; draft OFF per [amendment A4](#crescent-amendment-a4-a1-is-k3d-gpus-best-cpu-config-for-the-xeon-2026-10-20): H34x REFUTED it on this Xeon) still delivers far more B24 tasks within the 120 s turn budget than EAT stock B1 (vLLM CPU bf16, `xeon-values.yaml` as shipped, AMX on), both arms in one hold, interleaved. H49b's comparison moved to the hardware EAT is built for. A stack-vs-stack claim (amendment 16). | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement on the Xeon. A1 within-budget success >= 0.70 (0.8). B1 >= 0.60 (0.7): with AMX bf16 its turns should fit 120 s. Paired A1 - B1 in [-0.15, +0.20] (0.6). L2 matches >= 16 pairs and >= 75% (0.6), and the gates are read first, so the expected verdict is REFUTED (0.5), INCONCLUSIVE (0.45, mostly the L2 gates), SUPPORTED (0.05). Not gated: median attempt A1/B1 <= 0.7 (0.6). | H49b's design, statistic, outcome classes, grader, gates and thresholds on `crescent`; see [H50-H52 pre-registration](#h50-h52-pre-registration-the-xeon-6741p-2026-10-20). Holds of at most 40 min until 24 pairs or 8 holds. SUPPORTED: paired difference CI lower bound >= +30 points and A1 >= 0.6, with L2, the void and idle-arm checks passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. `bench/eatflow/crescent/pairs-hold.sh H50 ` (`LIVE_MODE=pairs LIVE_CAP=2370 live.sh H50`, KV 40 GiB), `agentbench pairs-report --hyp H50 --arm-a A1 --arm-b B1`. Model, arms, sampling and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20) (Gemma 4 26B-A4B QAT; breaks direct replication of H49b). Analysis rule L3 (primary) from [amendment A3](#crescent-amendment-a3-analysis-rule-l3-for-the-crescent-pairs-rows-2026-10-20). B1 is EAT release-1.1.0's stock (vLLM CPU v0.28.0) and runs only after a stall check, per [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); H50's deploy commit, wiring smoke and pre-hold records per [amendment E3](#crescent-amendment-e3-h50s-deploy-commit-wiring-smoke-and-pre-hold-records-2026-10-29). Result ([H50 run](#h50-run-xeon-6741p-2026-10-29)): E1's stall check passed (60 of 60 requests, no stall); 24 complete pairs in 1 of the 8 permitted holds, no void pair, idle check 0.0022 CPU, balance gate passed (smallest p 0.97). Within-budget success A1 19/24 (0.79), B1 21/24 (0.875); paired difference A1 - B1 -0.083 (95% CI -0.208 to 0.000), under +30 points, so REFUTED. Not gated: median attempt A1 8.0 s, B1 12.0 s; paired J per attempt A1/B1 0.81. | REFUTED | | H50o | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) H50's comparison with arm B3, EAT's own OVMS chart (`core/helm-charts/ovms` at release-1.0.3-hotfix, stock values, `openvino/model_server` 2025.4, the closest OpenVINO/ model it pulls, `OpenVINO/Qwen3-4B-int4-ov`) instead of B1: A1 delivers far more B24 tasks within budget than B3. With H50 it splits B1 -> A1 into EAT switching its own engine (B1 -> B3) and what k3d-gpu adds on top (B3 -> A1). | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement. B3's stock graph has no tool parser (the chart's `ovms --pull` writes none) and its model reasons by default, so its tool calls arrive as text: B3 within-budget success <= 0.10 (0.7), paired A1 - B3 >= +60 points (0.65). Pair yield: a B3 attempt should end within 30-90 s (a failed tool call or a reasoned text answer), so a 40-min hold should run 15-25 pairs and 24 pairs fit 2-3 holds; L2 passes (0.6), so SUPPORTED (0.5), INCONCLUSIVE (0.45). | As H50 with arm B = B3 (same pairs, gates, thresholds, statistic, cluster per hold `k3dgpu-e-h50o`); holds of at most 40 min until 24 pairs or 6 holds. The B1 -> B3 and B3 -> A1 splits are described from the two rows' matched pairs, never gated. `bench/eatflow/crescent/pairs-hold.sh H50o `, `agentbench pairs-report --hyp H50o --arm-a A1 --arm-b B3`. Model and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20). Analysis rule L3 from [amendment A3](#crescent-amendment-a3-analysis-rule-l3-for-the-crescent-pairs-rows-2026-10-20). B3 is unchanged under [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29) (EAT release-1.1.0's OVMS chart renders the same objects); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); H50o's deploy commit, B3's records and its wiring smoke per [H50o amendment O1](#h50o-amendment-o1-deploy-commit-b3s-records-and-a-b3-wiring-smoke-2026-10-29). Result ([H50o run](#h50o-run-xeon-6741p-2026-10-29)): B3 was not ready in hold 1, so under M1's rule H50o runs no further hold and no pair ran (1 of the 6 permitted holds). On k3d, B3's PVC bound, its pod was scheduled and its init container pulled `OpenVINO/gemma-4-26b-a4b-it-int4-ov` at `4e5885c3` (every file equal to the Hub's), but OVMS 2025.4's server failed on each of its 7 starts, going into CrashLoopBackOff; the last start's recorded exit is 139 (SIGSEGV), a second into loading the model. The same server in a plain container, outside k3d, on the same revision, exited 139 the same way with and without the chart's limits. Not deliverable: EAT's OVMS chart as shipped does not serve this model. | INCONCLUSIVE | | H51 | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) Graded agents per box on the Xeon 6741P: H45's ramp, gates and scoring (B15-RO, typical load, 300 s steps, pooled p95 turn <= 10 s, success >= the arm's N=1 success - 10 points, errors <= 1%, >= 60 window requests), extended to N=16, for A1 and B1. | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement. A1 in [1, 8] (0.5). B1 in [0, 4] (0.5). B1 <= A1 (0.55). | H45's `apb` runner and scorer (`apb-score --hyp H51`, ramp 1, 2, 4, 8, 16), one fresh cluster per hold, at most 4 holds of at most 40 min. A prediction holds when the arm's bracket lies inside its range and is refuted when it lies outside; B1 <= A1 is decided only on matched load. REFUTED when any prediction is refuted, SUPPORTED when all three hold, else INCONCLUSIVE. Runs only if H50's holds end within their cap. Model and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20). B1 as H50, from [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); deploy commit, M1's ranges in the scorer, one hold per arm and the wiring smokes per [H51 amendment S1](#h51-amendment-s1-deploy-commit-scorer-ranges-ramp-per-hold-and-wiring-smokes-2026-10-29). Result ([H51 run](#h51-run-xeon-6741p-2026-10-29)): one hold per arm, 2 of the 4 permitted, none VOID. Both arms failed N=1. A1: 65 window requests, turn p95 10.41 s against the 10 s SLO, success 30/32. B1: 58 window requests (under 60), p95 11.52 s, success 27/29. Agents per box A1 0, outside its predicted [1, 4], so REFUTED. B1's 0 lies inside [0, 4], and B1 <= A1 holds on matched load (0 <= 0). | REFUTED | -| H62 | Against Intel's new default. On the Xeon 6741P, inside EAT's banking flow (B24) with Gemma 4 26B-A4B on its QAT weights, the k3d-gpu CPU stack A1 (H50's A1 unchanged: `agent manifest`, OVMS 2026.4.1 int4 on the snapped QAT IR, CPU, no draft) delivers far more tasks within the 120 s turn budget than B5, Intel's documented single-node default for its agent toolkit: EAS v1.2.0-rc.1 (`5c27fb3`) installing EAT 2.0 (`release-2.0.0` `036b662`) and enterprise-inference `release-1.1` (`170eb3a`) with Kubespray onto one node, serving its catalog entry `gemma-4-26b` (vLLM CPU v0.28.0 on KServe, NRI balloons) on the same QAT weights. Each stack runs in its own KVM guest of one shape on the same host; both are live in one hold, interleaved. Deviations from stock are named (D1-D3). A stack-vs-stack claim (amendment 16). | Written 2026-10-29, before any H62 step and after H50 (A1 19/24, B1 21/24) and H51 were known. A1 within-budget success >= 0.70 (0.65). B5 >= 0.70 (0.6). Paired A1 - B5 in [-0.20, +0.15] (0.6). Balance gate passes (0.9). REFUTED (0.6), INCONCLUSIVE (0.35: mostly B5 not deliverable in its guest within 120 min, a failed wiring smoke or a gate), SUPPORTED (0.05). Not gated: median attempt A1/B5 <= 0.85 (0.5); A1's median attempt in its guest over H50's 8.0 s on the host in [1.0, 1.3] (0.6). | [H62 pre-registration](#h62-pre-registration-k3d-gpu-against-intels-new-default-eas-with-eat-20-on-the-xeon-6741p-2026-10-29): H50's design, statistic, gates and verdict rule (rule L3, amendment A3, seed `H62-crescent-2026-10-29`), two KVM guests on crescent (VM a 96 vCPU and 96 GiB for A1, VM b 96 vCPU and 192 GiB for B5), the harness's LiteLLM per arm on the host, holds of at most 40 min until 24 pairs or 8 holds, each hold restarting both serving pods. Setup and wiring-smoke gates (memory, disk, AMX in each guest, NFD label, B5's weights equal to the QAT checkpoint, EAS's own `validate` and gateway check) come before hold 1. SUPPORTED: paired difference CI lower bound >= +30 points and A1 >= 0.6, with every gate passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. `bench/h62/campaign.sh`; `agentbench pairs-report --hyp H62 --arm-a A1 --arm-b B5`. Re-run as a new row when EAS v1.2.x or an EAT 2.x release is tagged (`bench/h62/upstream.sh`). | PENDING | +| H62 | Against Intel's new default. On the Xeon 6741P, inside EAT's banking flow (B24) with Gemma 4 26B-A4B on its QAT weights, the k3d-gpu CPU stack A1 (H50's A1 unchanged: `agent manifest`, OVMS 2026.4.1 int4 on the snapped QAT IR, CPU, no draft) delivers far more tasks within the 120 s turn budget than B5, Intel's documented single-node default for its agent toolkit: EAS v1.2.0-rc.1 (`5c27fb3`) installing EAT 2.0 (`release-2.0.0` `036b662`) and enterprise-inference `release-1.1` (`170eb3a`) with Kubespray onto one node, serving its catalog entry `gemma-4-26b` (vLLM CPU v0.28.0 on KServe, NRI balloons) on the same QAT weights. Each stack runs in its own KVM guest of one shape on the same host; both are live in one hold, interleaved. Deviations from stock are named (D1-D4; D4: the harness reaches B5's workload Service, not EAS's LiteLLM gateway and guardrail). A stack-vs-stack claim (amendment 16). | Written 2026-10-29, before any H62 step and after H50 (A1 19/24, B1 21/24) and H51 were known. A1 within-budget success >= 0.70 (0.65). B5 >= 0.70 (0.6). Paired A1 - B5 in [-0.20, +0.15] (0.6). Balance gate passes (0.9). REFUTED (0.6), INCONCLUSIVE (0.35: mostly B5 not deliverable in its guest within 120 min, a failed wiring smoke or a gate), SUPPORTED (0.05). Not gated: median attempt A1/B5 <= 0.85 (0.5); A1's median attempt in its guest over H50's 8.0 s on the host in [1.0, 1.3] (0.6). | [H62 pre-registration](#h62-pre-registration-k3d-gpu-against-intels-new-default-eas-with-eat-20-on-the-xeon-6741p-2026-10-29): H50's design, statistic, gates and verdict rule (rule L3, amendment A3, seed `H62-crescent-2026-10-29`), two KVM guests on crescent (VM a 96 vCPU and 96 GiB for A1, VM b 96 vCPU and 192 GiB for B5), the harness's LiteLLM per arm on the host, holds of at most 40 min until 24 pairs or 8 holds, each hold restarting both serving pods. Setup and wiring-smoke gates (memory, disk, AMX in each guest, NFD label, B5's weights equal to the QAT checkpoint, EAS's own `validate` and gateway check) come before hold 1. SUPPORTED: paired difference CI lower bound >= +30 points and A1 >= 0.6, with every gate passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. `bench/h62/campaign.sh`; `agentbench pairs-report --hyp H62 --arm-a A1 --arm-b B5`. Re-run as a new row when an EAS release at v1.2.0 or later or an EAT release at 2.0.0 or later is tagged (`bench/h62/upstream.sh`). | PENDING | | H52 | A plain `k3d-gpu up` (no flags, the auto-adapt default) on the Xeon 6741P host as found detects and enables every engine actually present and advertises each usable one to the scheduler, with nothing missing and nothing phantom; engines present but not configured on the host are named, not advertised. | Written 2026-10-20 from an inventory read from sysfs, lspci and CPUID, not from k3d-gpu. 4 x Arc Pro B70 (xe) as `gpu.intel.com/xe: 4`; AMX TILE, BF16, INT8, FP16 and 14 AVX-512/VNNI labels; no hybrid, NUMA, NVIDIA, Gaudi or NPU objects; DSA, IAA and QAT named as present but unconfigured (0.4: `up` may not name them). | The table in [H50-H52 pre-registration](#h50-h52-pre-registration-the-xeon-6741p-2026-10-20): `k3d-gpu doctor`, `K3D_GPU_CLUSTER=k3dgpu-h52 k3d-gpu up`, `kubectl get node -o json`, `k3d-gpu test`, and one enumeration pod per B70. SUPPORTED: every row matches. REFUTED: any row does not. Run 2026-10-20 ([H52 run](#h52-run-xeon-6741p-2026-10-20)): the four B70s (`gpu.intel.com/xe: 4`, `xmx=true`), AMX (TILE, BF16, INT8, FP16) and all 14 AVX-512/VNNI labels detected and reached pods, nothing phantom; but no media engine label on `xe`, and `up` does not name the unconfigured DSA, IAA and QAT (`doctor` does). | REFUTED | | H52b | After `k3d-gpu host-setup --apply` configures the host's DSA, IAA and QAT (the fix that adds that path, merged first), a plain `k3d-gpu up` also advertises them, and each one reaches a pod; the host is then restored exactly. | Written 2026-10-20, before the fix exists. DSA, IAA and QAT allocatable > 0, each at the count host-setup reports configuring; their test pods see `/dev/dsa`, `/dev/iax`, `/dev/vfio` (0.6). The other rows read as H52's expected table, media labels included ([amendment D1](#h52-amendment-d1-the-fixes-and-h52b-2026-10-20)). | Runs after H52, once the host-setup fix is merged; same procedure plus `host-setup --apply` before `up` and its printed revert after. SUPPORTED: the three engines advertised and reached, the H52 rows unchanged, and the host inventory read back equal to the one taken before. REFUTED: any of these fails. The host-setup command (`--apply --with-kernel-wqs`), the plan read first, the counts, the plain `up`'s node image and the revert per [amendment D2](#h52-amendment-d2-h52bs-host-setup-step-on-crescent-2026-10-29); the runner and deploy commit per [amendment D3](#h52-amendment-d3-h52bs-runner-and-deploy-commit-2026-10-29). Result ([H52b run](#h52b-run-xeon-6741p-2026-10-29)): one hold at k3d-gpu 0.1.79; the plan passed D2's gate (8 commands) and applied. DSA advertised 3 of the 3 configured work queues and its pod saw `/dev/dsa`; IAA got no work queue and no resource (as D2 item 4 disclosed); QAT got 48 VFs, but its plugin crash-looped and advertised none; `gpu.intel.com/xe` read 2, not 4: the serving pairs hold about half the VRAM of all four cards, and `up`'s busy-card filter (half the VRAM) left out the two just over half (16.1, 16.2 GiB) and kept the two just under it (15.4, 15.5 GiB), which the pairs also hold, so two enumeration pods ran, not four. Media labels (`vcs` 2, `vecs` 2), AMX, the 14 AVX-512/VNNI labels, CPU 96 and no other vendor read as expected. The third inventory equals the first. Three clauses fail, so REFUTED. | REFUTED | | H53 | Inside EAT's banking flow on the Arc Pro B70s with Qwen3.8-27B (thinking off), k3d-gpu's own GPU choice AG (OVMS int4 on one card, `agent manifest`, auto device) delivers far more B24 tasks within the 120 s turn budget than B2, "EAT PR #21 nearest profile (Qwen3.6-27B) on Qwen3.8 weights, fp16" (vLLM XPU v0.28.0, TP 4). A stack-vs-stack claim. | Written 2026-10-20, before any H53 measurement. AG >= 0.70 (0.6); B2 boots by ladder rung (b) (0.6); B2 >= 0.50 (0.5); block mean AG - B2 in [-0.15, +0.30] (0.5). INCONCLUSIVE (0.6), REFUTED (0.3), SUPPORTED (0.1). | Solo holds in the order AG, B2, B2, AG, AG, B2 (up to 8), blocks = hold pairs (2k-1, 2k), block-level inference: mean of per-block d over >= 3 counted blocks, t-interval. SUPPORTED: lower bound >= +30 points and AG >= 0.6, L3 gates passed. REFUTED: upper bound < +30 points. Else INCONCLUSIVE. `bench/eatflow/crescent/solo-hold.sh N ARM`, `agentbench solo-report --hyp H53 --arm-a AG --arm-b B2`. See [H53 pre-registration](#h53-pre-registration-eats-banking-flow-on-arc-pro-b70-2026-10-20). | PENDING | @@ -4216,6 +4216,11 @@ never edited. A protocol change is a dated amendment, merged before the data it - EAT 2.0 has no installer of its own. EAT `release-2.0.0`'s README ("Installing this release") says the Agent Toolkit installs through the Intel AI for Enterprise Solutions (EAS) installer: `./es_auto_installer.sh configure`, `init agent-toolkit`, `install agent-toolkit`. +- Intel's three sources name three EAS clones. That README clones EAS with no `-b`, which is + main (`97f6157`), whose layer manifest names EI `main` and EAT branch `agentic-toolkit-migration` + (`fb59459`), not the release. `docs/single-node-deployment.md` clones `-b release-1.2`. The rc.1 + release notes clone `-b v1.2.0-rc.1` and say it replaces the guide's clone. B5 follows the + release notes; `release-1.2`'s head is the same commit (read 2026-10-29). - H62 asks H50's question against that path: Intel's documented single-node default, run as documented, at pinned commits. @@ -4286,8 +4291,13 @@ never edited. A protocol change is a dated amendment, merged before the data it - So B5 runs Intel's default path unmodified, on a supported OS, in a disposable KVM guest on crescent (Ubuntu 24.04), isolated from the host's Docker serving. - A1 runs in a guest of the same shape. A1 on bare metal against B5 in a guest would hand - k3d-gpu an edge that comes from virtualization, not from either stack. A1 in its guest - against H50's A1 on the host is reported as a description of the VM tax, never gated. + k3d-gpu an edge that comes from virtualization, not from either stack. One shape gives + neither arm bare metal; it does not show the guest costs both engines alike. vLLM CPU binds + one OpenMP thread per core and OVMS schedules its own pool, and both run on unpinned vCPUs. + So the VM tax is described on both sides, never gated: A1 in its guest against H50's A1 on + the host, and B5 in its guest against H50's B1 on the host (the same vLLM CPU v0.28.0 image + and QAT weights under another deployment, so that side reads the guest and the deployment + together). **Arms.** @@ -4307,7 +4317,7 @@ never edited. A protocol change is a dated amendment, merged before the data it in H50. Amendment A4's record runs once at setup: `k3d-gpu plan --device cpu --model google/gemma-4-26B-A4B-it --explain` in VM a. A configuration other than A1's is recorded and does not change A1. -- **B5**: Intel's documented single-node install in VM b, with deviations D1-D3 only. The steps, +- **B5**: Intel's documented single-node install in VM b, with deviations D1-D4 only. The steps, in order: 1. The node kernel settings of EAT's `docs/prerequisites.md` ("Node Kernel Settings"), verbatim: `fs.inotify.max_user_instances = 8192` and `fs.inotify.max_user_watches = 1048576`. @@ -4357,6 +4367,13 @@ never edited. A protocol change is a dated amendment, merged before the data it - QEMU 10.2 (`qemu-system-x86` 1:10.2.1+ds-1ubuntu3.3) runs in a container built from `ubuntu:26.04@sha256:6a06111c...`, with `--device /dev/kvm`, so the host gets no package. QEMU passes AMX through from 7.0 on, and the host kernel (7.0) supports AMX for KVM guests. +- **D4, the request path.** Intel's agents reach the model through EAS's LiteLLM gateway, with the + content-safety guardrail and the hybrid router (EAT `config.yaml` lines 20-28; the guide's + "Verify the Base Stack" calls `https://litellm./v1/chat/completions`). The + harness reaches B5's workload Service directly instead (below), so B5's turns skip that + gateway, the guardrail's Qwen3Guard check and the router. Each skipped hop is a step every + stock turn takes (the guardrail is a model call), so D4 leans toward B5. The guardrail stays + deployed and holds its CPUs. **The harness, shared by both arms, as in H50.** @@ -4366,10 +4383,10 @@ never edited. A protocol change is a dated amendment, merged before the data it - Each arm's server is reached through a `kubectl port-forward` of its Service inside its guest, forwarded to the host. For A1 that is `svc/ovms`. For B5 it is `gemma-4-26b-kserve-workload-svc:8000`, the Service EI's own LiteLLM bypass uses. -- B5's route therefore skips EAS's Envoy AI gateway, Istio, LiteLLM and the guardrail. H50 - compared engines behind one constant gateway, and with one serving replica the endpoint picker - has one endpoint. EAS's gateway and guardrail are exercised in the wiring smoke and are not - gated. +- B5's route therefore skips EAS's Envoy AI gateway, Istio, LiteLLM and the guardrail + (deviation D4). H50 compared engines behind one constant gateway, and with one serving + replica the endpoint picker has one endpoint. EAS's gateway and guardrail are exercised in + the wiring smoke and are not gated. **Model and sampling.** Amendment M1's, unchanged. Both arms serve the QAT-trained weights: A1 the snapped INT4 IR, B5 the bf16 safetensors. Sampling is profile `agent_gemma4` (temperature @@ -4400,8 +4417,9 @@ prompts fit both, and the difference is part of stock B5. - the period-balanced estimate and L2's matched subset; - the serving covariate, plus each guest's CPU-s/s (`serving.py`'s `vm_cpu`); - B5's rendered CPU request, cpuset, `nproc` and the ISA its pod sees, per hold; - - the VM tax: A1 in its guest against H50's A1 on the host, within-budget success and median - attempt (`h62.py vmtax`). + - the VM tax: A1 in its guest against H50's A1 on the host, and B5 in its guest against H50's + B1 on the host (guest and deployment together), within-budget success and median attempt + (`h62.py vmtax`). **Holds, caps and gates.** @@ -4466,7 +4484,9 @@ prompts fit both, and the difference is part of stock B5. **Risks, named before measuring.** - AMX passthrough and the VM tax are unmeasured on crescent. The guest gate catches the first; - the VM tax is described. + the VM tax is described for both arms. It need not be equal: vLLM CPU's per-core OpenMP + threads and OVMS's pool can respond differently to vCPU preemption on unpinned vCPUs, so a gap + between the two descriptions is reported beside the verdict. - Kubespray rc.1 on one node behind user-mode networking has never run here, and that networking carries roughly 60 GB of images and weights into VM b. - VM a reads A1's IR over virtfs (9p). If the cloud image's kernel lacks the 9p modules, @@ -4475,8 +4495,10 @@ prompts fit both, and the difference is part of stock B5. first; each hold records the pod's image ID for the second. - Qwen3Guard deploys first, so `cpu: auto` sizes gemma's balloon to the cores left after it. The deploy order and B5's rendered request and cpuset are recorded. -- Newer upstream releases not used, so stock pins stay: Kubespray v2.32.0 (stock v2.30.0) and vLLM - v0.32.0 (stock v0.28.0). +- Newer upstream releases not used, so stock pins stay: Kubespray v2.32.0 (stock v2.30.0), vLLM + v0.32.0 (stock v0.28.0) and KServe v0.21.0 (stock v0.18.0, EI `roles/kserve`). For the guests, + QEMU 11.0.3 is packaged only in Ubuntu series newer than crescent's 26.04, whose `-updates` + pocket carries 10.2.1. - `qemu-system-x86` 1:10.2.1+ds-1ubuntu3.3 is the archive's current `-updates` version. If it is superseded before the image is built, the build fails and an amendment re-pins. - Both guests' vCPUs float over all 96 host CPUs, the load CPUs included. VM b is never idle at @@ -4486,8 +4508,10 @@ prompts fit both, and the difference is part of stock B5. **Re-run trigger.** - `bench/h62/upstream.sh` (`h62.py trigger`) reads both repositories' tags. It fires on an EAS - v1.2.x release tag (not a release candidate) or an EAT 2.x release tag (`release-2.X.Y` or - `v2.X.Y`). It is run before setup and before each hold, and its output goes into the records. + release tag at v1.2.0 or later (`vX.Y.Z`, not a release candidate) or an EAT release tag at + 2.0.0 or later (`release-X.Y.Z`, its `-hotfix` form, or `vX.Y.Z`). Full support may ship under + a later number than v1.2.x or 2.x, so any later release fires. It is run before setup and + before each hold, and its output goes into the records. - Fired before hold 1: a dated amendment re-pins B5 to the tagged release, by amendment E1's logic (EAT stock is what Intel ships now). If the tag's components equal the pinned commits, the amendment only records that. diff --git a/bench/h62/h62.py b/bench/h62/h62.py index 88ced803..58529914 100644 --- a/bench/h62/h62.py +++ b/bench/h62/h62.py @@ -30,7 +30,8 @@ h62.py trigger EAS_TAGS EAT_TAGS the re-run trigger, from two `git ls-remote --tags` outputs h62.py pairs-left PAIRS_DIR B24 pairs still to run h62.py vmtax H50_ANALYSIS H62_ANALYSIS - A1 on the host (H50) and in its guest (H62), described + A1 on the host (H50) and in its guest (H62), and H50's B1 + on the host against B5 in its guest, described """ import difflib import json @@ -406,11 +407,19 @@ def _tags(ls_remote): return {m.group(1) for m in re.finditer(r"refs/tags/([^\s^]+)(?:\^\{\})?\s*$", ls_remote, re.M)} +def _release(tag, pattern, at_least): + m = re.fullmatch(pattern, tag) + return bool(m) and tuple(int(x) for x in m.groups()) >= at_least + + def trigger(eas_ls_remote, eat_ls_remote): - """The re-run trigger: an EAS v1.2.x release tag (not a release candidate), or an EAT 2.x - release tag (EAT tags its releases release-X.Y.Z; a vX.Y.Z tag counts too).""" - eas = sorted(t for t in _tags(eas_ls_remote) if re.fullmatch(r"v1\.2\.\d+", t)) - eat = sorted(t for t in _tags(eat_ls_remote) if re.fullmatch(r"(release-|v)?2\.\d+\.\d+", t)) + """The re-run trigger: an EAS release tag at v1.2.0 or later (vX.Y.Z, not a release + candidate), or an EAT release tag at 2.0.0 or later (EAT tags its releases release-X.Y.Z and + its hotfixes release-X.Y.Z-hotfix; a vX.Y.Z tag counts too). Full support may ship under a + later number than v1.2.x or 2.x, so any later release fires.""" + eas = sorted(t for t in _tags(eas_ls_remote) if _release(t, r"v(\d+)\.(\d+)\.(\d+)", (1, 2, 0))) + eat = sorted(t for t in _tags(eat_ls_remote) + if _release(t, r"(?:release-|v)?(\d+)\.(\d+)\.(\d+)(?:-hotfix)?", (2, 0, 0))) return {"eas": eas, "eat": eat} @@ -422,15 +431,26 @@ def pairs_left(pairs_dir, total=TASKS): def vmtax(h50, h62): - """A1 on the host (H50's analysis) against A1 in its guest (H62's): described, never gated.""" - def arm(a): - s = a["within_budget_success"]["A1"] - return {"success": f"{s['k']}/{s['n']}", "attempt_median_s": a["attempt_median_s"]["A1"]} - host, guest = arm(h50), arm(h62) - r = None - if host["attempt_median_s"] and guest["attempt_median_s"]: - r = round(guest["attempt_median_s"] / host["attempt_median_s"], 3) - return {"a1_host": host, "a1_guest": guest, "attempt_median_guest_over_host": r, "gated": False} + """Described, never gated. A1 on the host (H50's analysis) against A1 in its guest (H62's): + the same stack, so the guest alone. And H50's B1 on the host against B5 in its guest: the same + vLLM CPU v0.28.0 image and QAT weights under another deployment, so the guest and the + deployment together; it shows whether the guest costs vLLM's arm more than OVMS's.""" + def arm(a, name): + s = (a.get("within_budget_success") or {}).get(name) + if not s: + return None + return {"success": f"{s['k']}/{s['n']}", "attempt_median_s": (a.get("attempt_median_s") or {}).get(name)} + + def ratio(host, guest): + if host and guest and host["attempt_median_s"] and guest["attempt_median_s"]: + return round(guest["attempt_median_s"] / host["attempt_median_s"], 3) + return None + a_host, a_guest, b_host, b_guest = arm(h50, "A1"), arm(h62, "A1"), arm(h50, "B1"), arm(h62, "B5") + return {"a1_host": a_host, "a1_guest": a_guest, "attempt_median_guest_over_host": ratio(a_host, a_guest), + "b1_host": b_host, "b5_guest": b_guest, "b_attempt_median_guest_over_host": ratio(b_host, b_guest), + "b_note": "B1 (EAT release-1.1.0's chart) and B5 (EAS's KServe deployment) share the vLLM image and " + "weights, not the deployment: the guest and the deployment are read together", + "gated": False} def _load(path): diff --git a/bench/h62/test_h62.py b/bench/h62/test_h62.py index d17a0411..5c0f2c6c 100644 --- a/bench/h62/test_h62.py +++ b/bench/h62/test_h62.py @@ -467,7 +467,8 @@ def test_cli_writes_the_script(self): class TriggerTest(unittest.TestCase): - """The re-run trigger: EAS v1.2.0 (GA) tagged, or an EAT 2.x release tagged.""" + """The re-run trigger: an EAS release at v1.2.0 or later (no release candidate) tagged, or an + EAT release at 2.0.0 or later tagged.""" EAS = ("1d5686e75f8ec0b1edac206e9ca4db7ddde32250\trefs/tags/v1.2.0-rc.1\n" "5c27fb39a95eecec41aa4e29a1188984e9c726f3\trefs/tags/v1.2.0-rc.1^{}\n" @@ -485,6 +486,18 @@ def test_ga_or_an_eat_2_release_fires(self): for tag in ("release-2.0.0", "v2.0.0", "release-2.1.0"): self.assertEqual(h62.trigger(self.EAS, self.EAT + f"f\trefs/tags/{tag}\n")["eat"], [tag], tag) + def test_a_later_ga_fires_too(self): + # Full support may ship under a later number than v1.2.x or 2.x; it still fires. + for tag in ("v1.3.0", "v2.0.0", "v1.10.0"): + self.assertEqual(h62.trigger(self.EAS + f"e\trefs/tags/{tag}\n", self.EAT)["eas"], [tag], tag) + for tag in ("v1.3.0-rc.1", "v1.1.1", "v1.0.0", "v2.0.0-beta"): + self.assertEqual(h62.trigger(self.EAS + f"e\trefs/tags/{tag}\n", self.EAT)["eas"], [], tag) + # EAT tags its hotfix releases release-X.Y.Z-hotfix (release-1.0.3-hotfix). + for tag in ("release-3.0.0", "release-2.0.1-hotfix", "v3.1.0"): + self.assertEqual(h62.trigger(self.EAS, self.EAT + f"f\trefs/tags/{tag}\n")["eat"], [tag], tag) + for tag in ("release-1.2.0", "release-2.0.0-rc1", "release-0.5.0"): + self.assertEqual(h62.trigger(self.EAS, self.EAT + f"f\trefs/tags/{tag}\n")["eat"], [], tag) + def test_cli(self): tmp = tempfile.mkdtemp() self.addCleanup(shutil.rmtree, tmp) @@ -507,15 +520,34 @@ def test_pairs_left(self): self.assertEqual(h62.pairs_left(tmp), 14) self.assertEqual(run("pairs-left", tmp).stdout, "14\n") + H50 = {"within_budget_success": {"A1": {"k": 19, "n": 24, "rate": 19 / 24}, "B1": {"k": 21, "n": 24, "rate": 0.875}}, + "attempt_median_s": {"A1": 8.0, "B1": 12.0}} + H62 = {"within_budget_success": {"A1": {"k": 20, "n": 24, "rate": 20 / 24}, "B5": {"k": 18, "n": 24, "rate": 0.75}}, + "attempt_median_s": {"A1": 9.2, "B5": 15.0}} + def test_vmtax_is_a_description(self): - h50 = {"within_budget_success": {"A1": {"k": 19, "n": 24, "rate": 19 / 24}}, "attempt_median_s": {"A1": 8.0}} - h62r = {"within_budget_success": {"A1": {"k": 20, "n": 24, "rate": 20 / 24}}, "attempt_median_s": {"A1": 9.2}} - t = h62.vmtax(h50, h62r) + t = h62.vmtax(self.H50, self.H62) self.assertEqual(t["a1_host"], {"success": "19/24", "attempt_median_s": 8.0}) self.assertEqual(t["a1_guest"], {"success": "20/24", "attempt_median_s": 9.2}) self.assertAlmostEqual(t["attempt_median_guest_over_host"], 1.15) self.assertEqual(t["gated"], False) + def test_vmtax_describes_the_vllm_side_too(self): + # B5 in its guest against H50's B1 on the host: the same vLLM CPU v0.28.0 image and QAT + # weights under another deployment, so the guest and the deployment are read together. + t = h62.vmtax(self.H50, self.H62) + self.assertEqual(t["b1_host"], {"success": "21/24", "attempt_median_s": 12.0}) + self.assertEqual(t["b5_guest"], {"success": "18/24", "attempt_median_s": 15.0}) + self.assertAlmostEqual(t["b_attempt_median_guest_over_host"], 1.25) + self.assertIn("deployment", t["b_note"]) + self.assertEqual(t["gated"], False) + + def test_vmtax_without_a_b5_reading(self): + h62r = {"within_budget_success": {"A1": {"k": 20, "n": 24}}, "attempt_median_s": {"A1": 9.2}} + t = h62.vmtax(self.H50, h62r) + self.assertEqual(t["b5_guest"], None) + self.assertEqual(t["b_attempt_median_guest_over_host"], None) + def stub(path, body): with open(path, "w") as f: