From 11e076dbb87731dc79ed812caf9a5f85e6ed9fed Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Thu, 29 Oct 2026 12:34:29 -0700 Subject: [PATCH 01/21] bench(h54q): the H54q runner, amendment Q2 H54q had no runner, and amendment Q1 asks for one before its first hold. This adds it. - arm.sh gains an item mode, which hold.sh turns on for row q. Each arm answers its GSM8K items 16 in flight, using H54a's chat and grader. Every finished request is written as it lands. No request starts after an item stop 90 s before the hold's deadline. An item's first error re-asks it once; a second error counts as wrong. - When the server stops answering, the arm is failed and the run of errors is discarded. - OVQ, OVB and Lg apply the manifests H54a's scored runs applied, byte for byte. R and Vi derive from H54a's recorded B1g render. - qcampaign.sh plans each of the three holds from the records so far. - h54.py score-q pairs items 0-199 and bootstraps OVQ - R and OVB - OVQ (10,000 resamples, seed 1). - The tests cover the renders, the record rules, the plan, the client against fakes and a local stub server, and the verdicts. --- EXPERIMENTS.md | 113 ++++++++++++- bench/h54/arm.sh | 52 +++++- bench/h54/h54.py | 329 +++++++++++++++++++++++++++++++++++- bench/h54/hold.sh | 2 + bench/h54/q-smoke.jsonl | 2 + bench/h54/qcampaign.sh | 25 +++ bench/h54/reproduce.sh | 8 + bench/h54/test_h54.py | 361 +++++++++++++++++++++++++++++++++++++++- 8 files changed, 882 insertions(+), 10 deletions(-) create mode 100644 bench/h54/q-smoke.jsonl create mode 100755 bench/h54/qcampaign.sh diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 424b48ce..83c4e520 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -72,7 +72,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H52b | After `k3d-gpu host-setup --apply` configures the host's DSA, IAA and QAT (the fix that adds that path, merged first), a plain `k3d-gpu up` also advertises them, and each one reaches a pod; the host is then restored exactly. | Written 2026-10-20, before the fix exists. DSA, IAA and QAT allocatable > 0, each at the count host-setup reports configuring; their test pods see `/dev/dsa`, `/dev/iax`, `/dev/vfio` (0.6). The other rows read as H52's expected table, media labels included ([amendment D1](#h52-amendment-d1-the-fixes-and-h52b-2026-10-20)). | Runs after H52, once the host-setup fix is merged; same procedure plus `host-setup --apply` before `up` and its printed revert after. SUPPORTED: the three engines advertised and reached, the H52 rows unchanged, and the host inventory read back equal to the one taken before. REFUTED: any of these fails. The host-setup command (`--apply --with-kernel-wqs`), the plan read first, the counts, the plain `up`'s node image and the revert per [amendment D2](#h52-amendment-d2-h52bs-host-setup-step-on-crescent-2026-10-29). | PENDING | | H53 | Inside EAT's banking flow on the Arc Pro B70s with Qwen3.8-27B (thinking off), k3d-gpu's own GPU choice AG (OVMS int4 on one card, `agent manifest`, auto device) delivers far more B24 tasks within the 120 s turn budget than B2, "EAT PR #21 nearest profile (Qwen3.6-27B) on Qwen3.8 weights, fp16" (vLLM XPU v0.28.0, TP 4). A stack-vs-stack claim. | Written 2026-10-20, before any H53 measurement. AG >= 0.70 (0.6); B2 boots by ladder rung (b) (0.6); B2 >= 0.50 (0.5); block mean AG - B2 in [-0.15, +0.30] (0.5). INCONCLUSIVE (0.6), REFUTED (0.3), SUPPORTED (0.1). | Solo holds in the order AG, B2, B2, AG, AG, B2 (up to 8), blocks = hold pairs (2k-1, 2k), block-level inference: mean of per-block d over >= 3 counted blocks, t-interval. SUPPORTED: lower bound >= +30 points and AG >= 0.6, L3 gates passed. REFUTED: upper bound < +30 points. Else INCONCLUSIVE. `bench/eatflow/crescent/solo-hold.sh N ARM`, `agentbench solo-report --hyp H53 --arm-a AG --arm-b B2`. See [H53 pre-registration](#h53-pre-registration-eats-banking-flow-on-arc-pro-b70-2026-10-20). | PENDING | | H54a | On the Xeon 6741P, Gemma 4 26B-A4B (QAT weights) single-stream: the best optimized CPU arm, chosen by the pre-registered source and speed rule, decodes at N=1 at least 3x faster than EAT stock B1g (vLLM CPU v0.24.0 bf16, `xeon-values.yaml`'s Gemma entry on the QAT bf16 weights), with a 50-item quality spot-check no more than 10 points below B1g's. A microbench, not an agent claim. | Written 2026-10-20, before any H54 measurement. N=1 median decode tok/s: B1g in [4, 15] (0.6); Lg (llama.cpp CPU, Google QAT Q4_0) in [25, 70] (0.6); A1g (k3d-gpu auto, OVMS CPU, QAT-matched IR) in [15, 60] (0.5), A1g/A1o in [0.85, 1.02] (0.6); Vq (vLLM CPU v0.32.0 bf16) in [6, 18] (0.5), Vq-mtp/Vq in [1.4, 2.5] (0.5); Lg-mtp/Lg in [1.2, 2.2] (0.4); Sg (SGLang CPU AMX) deliverable (0.6). The chosen arm is Lg or Lg-mtp (0.55). B3g not deliverable (0.8). SUPPORTED (0.65), INCONCLUSIVE (0.25), REFUTED (0.1). Not gated: TTFT on a 1,000-token prompt, aggregate tok/s at N=4 and 16 (A1g N=16/N=1 <= 1.5, 0.5), J/token. | [H54 pre-registration](#h54-pre-registration-gemma-4-26b-a4b-qat-and-qwen38-microbenches-on-crescent-2026-10-20): `bench/llm_load.py` per arm (N=1: 8 requests, N=4: 16, N=16: 32, 256 tokens, greedy, `ignore_eos`), `bench/energy.sh` RAPL per phase, GSM8K items 0-49 graded by exact match, one arm serving at a time, at most 3 holds of at most 40 min. Statistic: ratio of the chosen arm's median N=1 decode to B1g's, 95% bootstrap CI over requests. SUPPORTED: CI lower bound >= 3.0 and the quality gate passed. REFUTED: CI upper bound < 3.0, or every arm with a point ratio >= 3.0 fails the quality gate. Otherwise INCONCLUSIVE (also: B1g not deliverable, or a void phase not re-run). Run 2026-10-21, 3 holds ([H54a run](#h54a-run-xeon-6741p-2026-10-21)): stock B1g stalled in both its holds (vLLM v0.24.0 `EngineDeadError` mid-decode, cause open; no scored phase, 0/50 GSM8K, all errored), so `h54.py score-a` has no base and no chosen arm. Measured: Lg 48.74, Lg-mtp 54.52, A1g 23.25, A1o 21.73 tok/s N=1 median, GSM8K 50, 50, 49, 49 of 50; each scored phase exists only under amendment R4 (its own foreign CPU 4.4-4.8 CPU-s/s). Vq, Vq-mtp, Vi, Sg, B3g not run (cap). Descriptive, not scored: Lg-mtp over B1g's 2 hold-2 rates 3.02 ~~(CI 2.90-3.06)~~ (descriptive only: n=2 base rates from an engine that died minutes later, no CI; correction 2026-10-28, [H54a run](#h54a-run-xeon-6741p-2026-10-21)). | INCONCLUSIVE | -| H54q | QAT matters at int4 on this model: on GSM8K items 0-199, the QAT-matched OpenVINO IR OVQ (INT4_SYM, group 32, built from Google's QAT bf16 weights) is non-inferior to the bf16 QAT reference R within 5 points, and the published base-model IR OVB (`OpenVINO/gemma-4-26b-a4b-it-int4-ov`; ~~INT4_ASYM group 128~~ AWQ INT4 asymmetric group 64, router layers unquantized, per its `openvino_config.json`: [note N1](#h54-note-n1-what-ovb-is-2026-10-21)) scores below OVQ. | Written 2026-10-20, before any measurement. R accuracy in [0.85, 0.96] (0.6). OVQ - R in [-0.03, +0.02] (0.6). GGUF Q4_0 (Lg) - R in [-0.03, +0.02] (0.6). OVB - R in [-0.07, 0] (0.5). The OVB < OVQ part resolves on 200 items (0.2). INCONCLUSIVE (0.65), SUPPORTED (0.2), REFUTED (0.15). | As H54a's quality protocol on items 0-199 of `openai/gsm8k` `main` test @ `740312ad`; items 0-49 come from H54a's runs of the same arm configs; ~~at most 1 hold of at most 40 min for the rest~~ the rest in at most 3 holds of at most 40 min, R answering all 200 items because H54a never ran its config ([amendment Q1](#h54-amendment-q1-h54qs-reference-arm-hold-cap-and-order-2026-10-29)). Paired bootstrap 95% CI over items (10,000 resamples, seed 1). SUPPORTED: CI lower bound of OVQ - R >= -0.05 and CI upper bound of OVB - OVQ < 0. REFUTED: CI upper bound of OVQ - R < -0.05, or CI lower bound of OVB - OVQ > 0. Otherwise INCONCLUSIVE. | PENDING | +| H54q | QAT matters at int4 on this model: on GSM8K items 0-199, the QAT-matched OpenVINO IR OVQ (INT4_SYM, group 32, built from Google's QAT bf16 weights) is non-inferior to the bf16 QAT reference R within 5 points, and the published base-model IR OVB (`OpenVINO/gemma-4-26b-a4b-it-int4-ov`; ~~INT4_ASYM group 128~~ AWQ INT4 asymmetric group 64, router layers unquantized, per its `openvino_config.json`: [note N1](#h54-note-n1-what-ovb-is-2026-10-21)) scores below OVQ. | Written 2026-10-20, before any measurement. R accuracy in [0.85, 0.96] (0.6). OVQ - R in [-0.03, +0.02] (0.6). GGUF Q4_0 (Lg) - R in [-0.03, +0.02] (0.6). OVB - R in [-0.07, 0] (0.5). The OVB < OVQ part resolves on 200 items (0.2). INCONCLUSIVE (0.65), SUPPORTED (0.2), REFUTED (0.15). | As H54a's quality protocol on items 0-199 of `openai/gsm8k` `main` test @ `740312ad`; items 0-49 come from H54a's runs of the same arm configs; ~~at most 1 hold of at most 40 min for the rest~~ the rest in at most 3 holds of at most 40 min, R answering all 200 items because H54a never ran its config ([amendment Q1](#h54-amendment-q1-h54qs-reference-arm-hold-cap-and-order-2026-10-29); runner: [amendment Q2](#h54-amendment-q2-h54qs-runner-2026-10-29)). Paired bootstrap 95% CI over items (10,000 resamples, seed 1). SUPPORTED: CI lower bound of OVQ - R >= -0.05 and CI upper bound of OVB - OVQ < 0. REFUTED: CI upper bound of OVQ - R < -0.05, or CI lower bound of OVB - OVQ > 0. Otherwise INCONCLUSIVE. | PENDING | | H54b | On Arc Pro B70 with Qwen3.8-27B single-stream: the best optimized arm on ONE card (k3d-gpu auto OVMS int4, Intel's llm-scaler vLLM, or llama.cpp SYCL, with and without each one's documented MTP) decodes at N=1 at least 2x faster than EAT #21's XPU path B2q on TWO cards (vLLM XPU v0.28.0, the `xpu-qwen36-27b` profile's flags on the Qwen3.8-27B weights, TP 2). | Written 2026-10-20, before any measurement. N=1 median decode tok/s: X1q (OVMS GPU int4) in [15, 28] (0.5), X1q-mtp/X1q in [1.3, 2.2] (0.4); Sq (llm-scaler 1.5, online FP8, TP 1) deliverable (0.5), in [10, 18]; Sq-mtp/Sq in [1.4, 2.3] (0.5); Lsq (llama.cpp SYCL, Intel Q4_K_M) in [12, 24] (0.5); B2q deliverable at TP 2 (0.4), in [5, 12] (0.5). Chosen/B2q >= 2 (0.6). Cards free for a window (0.5). INCONCLUSIVE (0.6), SUPPORTED (0.3), REFUTED (0.1). | H54a's protocol on the B70s, cards only when `~/k3dgpu-bench/gpu-state.txt` lists them free; at most 2 holds of at most 40 min. Same statistic and quality gate (against B2q, or the official-weights arm Sq when B2q is not deliverable, for the gate only). SUPPORTED: CI lower bound of chosen one-card arm / B2q >= 2.0 and the gate passed. REFUTED: CI upper bound < 2.0. Otherwise INCONCLUSIVE (also: B2q not deliverable, or no free card before the campaign ends). Gemma 4 QAT on one B70 (X1g, X1o, B2g, Sgg) is described if hold time remains, never gated. Run 2026-10-21, 2 holds ([H54b and H54c run](#h54b-and-h54c-run-arc-pro-b70-2026-10-21)): INCONCLUSIVE, not delivered as pre-registered: the pinned "one card" pods saw all 4 cards. B2q, Sq, Sq-mtp (Qwen bf16, 80.6 GB with its guard) and Rq were not deliverable (disk); X1q and X1q-mtp failed on two `h54b.sh` runner bugs before any engine started; Lsq ran, but its pod listed every render node and llama.cpp split the model over all four B70s, two of them serving the live `qwen38-a-hr`, so no one-card arm exists. Lsq-mtp was not run: its GGUF carries the MTP (`nextn`) tensors, but G1's runner has no such arm; nor were the described-only Gemma arms. `score.py`: "no gate reference (B2q and Sq missing)". Descriptive only, a 4-card run and not a one-card or B70 result: Lsq 27.3 tok/s N=1 median, GSM8K 48/50. | INCONCLUSIVE | | H54c | Hybrid placement ("use the fast DRAM"): llama.cpp with Gemma 4 26B-A4B QAT Q4_0, routed experts on the Xeon and everything else on one B70 (Hg, `-ngl 99 -ot exps=CPU`), decodes at N=1 within 25% of a pre-stated byte-split model built from the same image's full-CPU (Cg) and full-GPU (Gg) runs. Qwen3.8-Flash-Next falls back to Gemma 4 QAT (reasons in the pre-registration). | Written 2026-10-20, before any measurement. Model: t_Hg = t_Cg x (B_exp / B_tot) + t_Gg x (B_rest / B_tot) + 60 x 30 us, bytes from the GGUF tensor table. Within 25% (0.45). Hg < Gg (0.8). Hg / Cg in [0.8, 1.6] (0.5). Gg in [30, 90] tok/s (0.4). SUPPORTED (0.45), INCONCLUSIVE (0.3), REFUTED (0.25). | One hold of at most 40 min on one free card: Cg, Gg, Hg in that order, then Gg again (drift check), all in `ghcr.io/ggml-org/llama.cpp:server-intel-b11749`, H54a's N=1 phase only. SUPPORTED: abs(Hg measured / predicted - 1) <= 0.25. REFUTED: > 0.40. Otherwise INCONCLUSIVE (also: any of the three arms not delivered, or the two Gg runs differ by > 10%). Run 2026-10-21, 1 hold ([H54b and H54c run](#h54b-and-h54c-run-arc-pro-b70-2026-10-21)): INCONCLUSIVE, not delivered as pre-registered: the pinned "one card" pods saw all 4 cards. Gg, Hg and Gg2 split their GPU layers over four B70s (two serving the live `qwen38-a-hr`), and Cg (`-ngl 0`) also allocated about 0.5 GB on one of those cards, so t_Cg and the prediction are confounded. The verdict does not come from `predict.py`'s band test, whose inputs are these 4-card and confounded runs. Descriptive only, not one-card or hybrid results: Cg 36.8, Gg 58.0, Hg 56.3, Gg2 57.4 tok/s N=1 median. | INCONCLUSIVE | | H54d | Inside EAT's banking flow with Gemma 4 26B-A4B (QAT), the best H54a CPU arm A* delivers far more B24 tasks within budget than EAT stock B1g, under H50's design, statistic, gates and thresholds (rule L2, amendment C1). Runs only when A* is not H50's A1 config. | Written 2026-10-20, before any measurement. A* within-budget success >= 0.60 (0.55). B1g >= 0.50 (0.6). Paired A* - B1g in [-0.10, +0.30] (0.6). A* serves B24's tool calls without a parser change (0.6). Fewer than 16 matched pairs in 3 holds (0.5). REFUTED (0.4), INCONCLUSIVE (0.5), SUPPORTED (0.1). | H50's procedure with arm A = A*, arm B = B1g; at most 3 holds of at most 40 min, until 24 pairs. SUPPORTED: paired difference CI lower bound >= +30 points and A* >= 0.6, L2, void and idle-arm checks passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. Not run (and so recorded) when A* is H50's A1 config: H50 answers it. Not run 2026-10-21, no hold used ([H54d: not run](#h54d-not-run-2026-10-21)): A* is undefined (H54a chose no arm), B1g stalled in both H54a holds, and H54q has not run; the caps rule makes a row with its stock or chosen arm missing INCONCLUSIVE. | INCONCLUSIVE | @@ -4659,6 +4659,117 @@ regression tests, merged before its first hold. fix, so the v0.24.0 stall of H54 note N2 (PR #209) is not expected in R. The image is not on crescent; the hold's cluster pulls it, as for every H54 arm. +### H54 amendment Q2: H54q's runner (2026-10-29) + +Recorded 2026-10-29, before any H54q hold or smoke. This is the runner amendment +[amendment Q1](#h54-amendment-q1-h54qs-reference-arm-hold-cap-and-order-2026-10-29) asks for. +Mechanics only. Q1's arms, models, item set, prompt, grader, statistic, thresholds, cap, order and +predictions do not change. + +**Files.** `bench/h54/h54.py` gains the `q-*` commands and `score-q`. `arm.sh` gains an item mode, +which `hold.sh` turns on for row `q` and its smoke row `smokeq`. `reproduce.sh` adds the sha256 of +each H54q manifest. `qcampaign.sh` runs the holds. `q-smoke.jsonl` holds the smoke's two +non-GSM8K questions. `test_h54.py` covers it all and already runs in CI. + +**Configs.** + +- OVQ, OVB and Lg apply the manifests H54a's scored runs applied, byte for byte: + `results/2026-10-21/H54a/hold3/A1g/deploy.yaml`, `hold3/A1o/deploy.yaml` and + `hold2/Lg/deploy.yaml`. + - These are not rendered again because `agent manifest` sizes OVMS from the host's memory at + render time. A1o's `--max_num_seqs=55`, `--cache_size=285` and 333Gi limit came from + crescent on 2026-10-21. A render on another day can write other values, and Q1 asks for the + same configs. + - Rendered on 2026-10-29 away from crescent, A1g's manifest was unchanged and A1o's was not. +- R and Vi run `derive_vllm`, unchanged since R2, on H54a's recorded B1g render + (`hold2/B1g/deploy.yaml`, equal to hold 1's). That is the transform `arm.sh` applies to a fresh + B1g render, without running EAT's chart through helm again. + - R's manifest is byte-identical to the Vq manifest that H54a's second wiring smoke booted on + 2026-10-21 (ready in 114 s). +- The cluster is H54a's: `hold-run.sh` is unchanged, with the 0.1.63 node image, AMX only, C1's + kubelet thresholds and the shim's model volume. The tools are the H54a runner's + (`~/k3dgpu-gemma/h54a/bin`, `~/k3dgpu-gemma/bin`: k3d v5.9.0 behind the shim, kubectl v1.35.2). + No arm is rendered with helm. + +**One arm's turn in a hold (`arm.sh` with `H54Q`).** + +- Times: + - The item stop is 90 s before the hold's arm deadline (2,370 s after the hold starts). + - An arm starts only with at least 300 s before the item stop. Otherwise it is not reached in + that hold. + - Readiness is the model list and a 16-token chat, polled every 10 s up to the arm's limit + (1,500 s for R, 900 s for the others), cut 60 s before the item stop. Past the limit the arm + is not deliverable, or not reached when the cut ended it. + - These replace R1's 900 s and 600 s. R1 sized them for an arm's full phase set; H54q keeps + items one by one across holds (Q1 item 4), so the end of a hold is not wasted on an arm that + cannot finish. +- Items: the arm's items still to answer, items that errored once first, then the rest in item + order. + - Asked 16 in flight with H54a's chat and grader, unchanged: prompt, temperature 0, + `max_tokens` 1024, thinking off, and the last number against the `####` answer. + - Each finished request is written to `items.jsonl` at once. + - No request starts after the item stop. A request still running then is cut and leaves no + row, so its item is not answered. +- Errors (Q1 item 5): an item's first error puts it back at the end of the same turn. If the + turn ends first, it heads the arm's next turn. A second error is recorded as a wrong answer. +- A server that stops answering (Q1 item 4, R3): + - The server is probed (model list and a 16-token chat, 300 s) in three cases: after 16 errors + in a row, after 600 s with requests in flight and none finishing, and when a turn's queue + ends on an error. A probe ends 5 s before the item stop, and none starts with less than 30 s + left. + - A failed probe ends the turn. The arm is failed with "server stopped answering", and the + errors since its last answer are discarded, so those items stay not answered. + - After a passing probe the errors stand. +- The item window's void check is recorded, not gated. H54a never voided quality items, and no + H54q statistic is timed. Teardown is R1's. + +**Holds (`qcampaign.sh`).** + +- Before each hold, `h54.py q-plan` reads the records so far. It names the arms with items left, + in Q1's order R, OVQ, OVB, Lg, Vi. + - An arm not deliverable or failed in two holds is not run again (R3 carries once). + - Vi, the probe-first arm, is not run again after its first loss. +- The hold then runs with `hold.sh q `. The lock, the H50 hold-start rule, the 97% + disk line and `timeout -k 30 2400` are unchanged. +- The campaign stops early when no arm has items left. Each `hold.sh` run counts toward Q1's cap + of 3. + +**Wiring smoke.** One smoke, before hold 1, under the lock: row `smokeq`, cluster +`k3dgpu-g-h54smokeq-1`, records under `~/k3dgpu-gemma/smoke/`. + +- R, OVQ and Lg each go through readiness, then answer `q-smoke.jsonl`'s two non-GSM8K questions. +- No GSM8K item is asked. +- It is discarded and disclosed in the run section, as H54a's three smokes were. It is not a hold. + +**Scorer (`h54.py score-q`).** + +- Each arm's items 0-199 come from H54a's records (OVQ, OVB and Lg, items 0-49) and from the + H54q rows. The first answer counts, a second error is wrong, and discarded rows are left out. +- A test exists only when both of its arms answered all 200 items. +- The paired bootstrap: d_i = a_i - b_i per item; 10,000 resamples of the 200 items with + replacement (Python's `random.Random(1).choices`); each test seeded 1 on its own. The CI is the + sorted means' entries 250 and 9,749, the same percentile indices as H54a's `boot_ratio`. +- The verdict, as the row states: + - REFUTED when a present test crosses a REFUTED bound. + - SUPPORTED when both tests are present and both SUPPORTED bounds hold. + - Otherwise INCONCLUSIVE. +- Lg - R, OVB - R and Vi - R are computed the same way and reported, not gated. + +**Records.** `~/k3dgpu-gemma/results/H54q/hold//` holds `items.jsonl`, `q.json`, +`status.json`, `deploy.yaml`, `server.log` and the readiness files. Each hold also keeps +`reproduce.txt`, `serving.jsonl`, `pre.txt` and `post.txt`. The campaign keeps `plan-hold.txt`, +`not-run.txt` and `score-q.json`. In the repository they go under +`bench/h54/results/2026-10-29/H54q/`. + +**Deploy commit.** The holds run from this amendment's branch head, pushed before the smoke and +hold 1, in a clean, detached checkout at `~/k3dgpu-gemma/h54q/repo`. It is not merged when they +run. [Amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29) +item 2 asks for `origin/main` at or after the runner amendment. This is the same deviation H50's +E3 and H51's S1 disclosed. + +**Predictions.** None is edited. Written with the H54q row's predictions in view and before any +H54q data existed. H54a's items 0-49 for OVQ, OVB and Lg were already known (49, 49 and 50 of 50). + ### H54a run (Xeon 6741P, 2026-10-21) ```text diff --git a/bench/h54/arm.sh b/bench/h54/arm.sh index e7783c93..58ee2076 100755 --- a/bench/h54/arm.sh +++ b/bench/h54/arm.sh @@ -7,6 +7,9 @@ # CLUSTER the hold's cluster; KUBECONFIG its scratch kubeconfig # DEADLINE epoch second the hold's arms must end by # SERVING the hold's serving.jsonl (bench/eatflow/crescent/serving.py) +# H54Q set: an H54q arm (amendment Q2): ARM is R, OVQ, OVB, Lg or Vi, its manifest comes +# from H54a's records (h54.py q-render), and after readiness it answers its GSM8K +# items (h54.py q-items) until the item stop, 90 s before DEADLINE; no phases # status.json: measured | not_deliverable | not_reached | failed, with the reason. set -uo pipefail ARM=$1 OUT=$2 @@ -19,8 +22,10 @@ now() { date +%s; } stamp() { echo "$(date -u +%FT%TZ) ${ARM} $*" | tee -a "${OUT}/arm.log"; } status() { ${PY} -c 'import json,sys; json.dump({"status": sys.argv[1], "reason": sys.argv[2], "at": sys.argv[3]}, open(sys.argv[4], "w"))' \ "$1" "$2" "$(date -u +%FT%TZ)" "${OUT}/status.json"; stamp "status $1: $2"; } -field() { ${PY} "${H}/h54.py" arms | ${PY} -c 'import json,sys; a=json.load(sys.stdin); print(a["arms"][sys.argv[1]].get(sys.argv[2], ""))' "${ARM}" "$1"; } -served() { ${PY} "${H}/h54.py" arms | ${PY} -c 'import json,sys; print(json.load(sys.stdin)["served"][sys.argv[1]])' "${ARM}"; } +B=${ARM} # the H54a arm whose server this is +[ -z "${H54Q:-}" ] || B=$(${PY} "${H}/h54.py" q-base "${ARM}") || exit 1 +field() { ${PY} "${H}/h54.py" arms | ${PY} -c 'import json,sys; a=json.load(sys.stdin); print(a["arms"][sys.argv[1]].get(sys.argv[2], ""))' "${B}" "$1"; } +served() { ${PY} "${H}/h54.py" arms | ${PY} -c 'import json,sys; print(json.load(sys.stdin)["served"][sys.argv[1]])' "${B}"; } NODE=k3d-${CLUSTER:?}-server-0 NSP=$(field ns) SVC=$(field svc) SPORT=$(field port) API=$(field api) BOOT=$(field boot) DEPLOY=$(field deploy) MODEL=$(served) @@ -40,9 +45,18 @@ pf_loop() { cleanup_pf() { [ -z "${pf}" ] || kill "${pf}" 2>/dev/null; pf=""; } trap cleanup_pf EXIT -left=$(( DEADLINE - $(now) )) -if [ "${left}" -lt 900 ]; then status not_reached "${left} s of the hold left (< 900)"; exit 0; fi -stamp "start (${left} s of the hold left)" +# H54a: an arm starts with at least 900 s of the hold left, and readiness ends 600 s before the +# hold does. H54q (Q2): no item is asked after STOP; an arm starts with at least 300 s before +# STOP, and readiness ends 60 s before it. +if [ -n "${H54Q:-}" ]; then + STOP=$(( DEADLINE - 90 )) + END=${STOP} NEED=300 EDGE=$(( STOP - 60 )) WHAT="before the item stop" +else + END=${DEADLINE} NEED=900 EDGE=$(( DEADLINE - 600 )) WHAT="of the hold left" +fi +left=$(( END - $(now) )) +if [ "${left}" -lt "${NEED}" ]; then status not_reached "${left} s ${WHAT} (< ${NEED})"; exit 0; fi +stamp "start (${left} s ${WHAT})" # ---- render -------------------------------------------------------------------------------- M=${OUT}/deploy.yaml @@ -57,7 +71,9 @@ eat_vllm() { # B1g: EAT's chart at 0d3438d as its playbook installs cpu-gemma4- --set ingress.enabled=false --set route.enabled=false > "${OUT}/eat.yaml" || return 1 ${PY} "${REPO}/bench/apb/stage.py" vllm "${OUT}/eat.yaml" "$1" google_gemma-4-26B-A4B-it-qat-q4_0-unquantized 8192 } +[ -z "${H54Q:-}" ] || DEPLOY=q case ${DEPLOY} in + q) ${PY} "${H}/h54.py" q-render "${ARM}" "${M}" ;; eat-vllm) eat_vllm "${M}" ;; derive-vllm) eat_vllm "${OUT}/b1g.yaml" && ${PY} "${H}/h54.py" derive-vllm "${ARM}" "${OUT}/b1g.yaml" "${M}" ;; eat-ovms) @@ -75,7 +91,7 @@ kubectl apply -f "${M}" > "${OUT}/apply.log" 2>&1 || { status failed "apply fail # ---- readiness: the model list and a 16-token chat ------------------------------------------ t_apply=$(now) -limit=$(( BOOT < DEADLINE - t_apply - 600 ? BOOT : DEADLINE - t_apply - 600 )) +limit=$(( BOOT < EDGE - t_apply ? BOOT : EDGE - t_apply )) base="http://127.0.0.1:${PORT}${API}" ready="" while [ $(( $(now) - t_apply )) -lt "${limit}" ]; do @@ -133,6 +149,30 @@ kubectl -n "${NSP}" get pod "${pod}" -o json > "${OUT}/pod.json" kubectl -n "${NSP}" exec "${pod}" -c "$(kubectl -n "${NSP}" get pod "${pod}" -o jsonpath='{.spec.containers[0].name}')" -- \ sh -c 'grep -c amx_tile /proc/cpuinfo; nproc' > "${OUT}/isa.txt" 2>&1 +if [ -n "${H54Q:-}" ]; then + # H54q (Q2): the arm's GSM8K items, or in a wiring smoke two non-GSM8K questions; no phases. + # The item window's void check is recorded, not gated (H54a never voided its quality items). + t0=$(date +%s.%N) + if [ -n "${SMOKE:-}" ]; then + ${PY} "${H}/h54.py" q-smoke "${base}" "${MODEL}" "${OUT}" "${STOP}" >> "${OUT}/arm.log" 2>&1 + status smoke "ready; wiring check in smoke-items.jsonl (rc $?)" + else + stamp "items: $(${PY} "${H}/h54.py" q-todo "$(dirname "$(dirname "${OUT}")")" "${ARM}" | wc -w) to ask" + ${PY} "${H}/h54.py" q-items "${base}" "${MODEL}" "$(dirname "$(dirname "${OUT}")")" "${ARM}" "${OUT}" "${STOP}" >> "${OUT}/arm.log" 2>&1 + rc=$? + if [ -s "${OUT}/q.json" ]; then + status "$(${PY} -c 'import json,sys; print(json.load(open(sys.argv[1]))["status"])' "${OUT}/q.json")" \ + "$(${PY} -c 'import json,sys; print(json.load(open(sys.argv[1]))["reason"])' "${OUT}/q.json")" + else + status failed "q-items exited ${rc} without a record" + fi + fi + t1=$(date +%s.%N) + echo "{\"phase\": \"items\", \"t0\": ${t0}, \"t1\": ${t1}, \"check\": $(${PY} "${H}/h54.py" void "${SERVING:?}" "${t0}" "${t1}")}" >> "${OUT}/phases.jsonl" + teardown + exit 0 +fi + if [ -n "${SMOKE:-}" ]; then # A wiring smoke (discarded, disclosed): one streamed 16-token request with ignore_eos, and one # chat with chat_template_kwargs, then the arm is deleted. No scored request. diff --git a/bench/h54/h54.py b/bench/h54/h54.py index 9c083a37..4a9462f8 100644 --- a/bench/h54/h54.py +++ b/bench/h54/h54.py @@ -17,8 +17,19 @@ h54.py carry RESULTS_DIR HOLD ARM... the arms HOLD carries to the next hold's front h54.py recheck RESULTS_DIR apply `served` to every arm recorded measured -Python 3 standard library only, except render/derive-vllm (PyYAML).""" - +H54q (amendments Q1 and Q2): quality only, items scored one by one and paired by item. + h54.py q-base ARM the H54a arm whose config an H54q arm runs + h54.py q-render ARM OUT.yaml the arm's manifest, from H54a's records + h54.py q-todo RESULTS_DIR ARM the items ARM still has to answer, in order + h54.py q-items URL MODEL RESULTS_DIR ARM OUT STOP + ask them, 16 in flight, until epoch STOP + h54.py q-smoke URL MODEL OUT STOP the wiring smoke's two non-GSM8K questions + h54.py q-plan RESULTS_DIR the arms the next hold runs, in order + h54.py score-q RESULTS_DIR the H54q tests and verdict + +Python 3 standard library only, except render/derive-vllm/q-render (PyYAML).""" + +import collections import concurrent.futures import glob import json @@ -28,6 +39,7 @@ import statistics import struct import sys +import threading import time import urllib.request @@ -505,6 +517,298 @@ def score_a(results): return rep +# ---- H54q (amendment Q1, runner amendment Q2) -------------------------------------------------- +# Each H54q arm runs an H54a config. OVQ, OVB and Lg apply the manifest H54a's scored run applied, +# byte for byte; R and Vi derive from H54a's recorded B1g render as H54a's arm.sh does. Items 0-49 +# of OVQ, OVB and Lg are H54a's records of those runs (Q1 item 2). + +HERE = os.path.dirname(os.path.abspath(__file__)) +H54A = os.path.join(HERE, "results", "2026-10-21", "H54a") +Q_N = 200 +Q_ORDER = ["R", "OVQ", "OVB", "Lg", "Vi"] +Q_ARMS = { + "R": dict(base="Vq", first=0, derive="hold2/B1g/deploy.yaml"), + "OVQ": dict(base="A1g", first=50, copy="hold3/A1g/deploy.yaml", h54a="hold3/A1g"), + "OVB": dict(base="A1o", first=50, copy="hold3/A1o/deploy.yaml", h54a="hold3/A1o"), + "Lg": dict(base="Lg", first=50, copy="hold2/Lg/deploy.yaml", h54a="hold2/Lg"), + "Vi": dict(base="Vi", first=0, derive="hold2/B1g/deploy.yaml", probe=True), +} +LOST = ("not_deliverable", "failed") # statuses of a hold in which the arm did not serve its turn + + +def q_render(arm, out, h54a=H54A): + a = Q_ARMS[arm] + if "copy" in a: + with open(os.path.join(h54a, a["copy"]), "rb") as f, open(out, "wb") as g: + g.write(f.read()) + return + import yaml + with open(os.path.join(h54a, a["derive"])) as f: + docs = derive_vllm(a["base"], [d for d in yaml.safe_load_all(f) if d]) + with open(out, "w") as f: + yaml.safe_dump_all(docs, f, sort_keys=False) + + +def holds(results): + """The hold numbers recorded under RESULTS_DIR, ascending.""" + out = [] + for d in glob.glob(os.path.join(results, "hold*")): + tail = os.path.basename(d)[4:] + if tail.isdigit(): + out.append(int(tail)) + return sorted(out) + + +def q_rows(path): + """(rows, discarded seqs) of one items.jsonl: the rows of finished requests, and the seqs a + `stopped` event discards.""" + rows, discard = [], set() + if not os.path.exists(path): + return rows, discard + with open(path) as f: + for line in f: + if not line.strip(): + continue + try: + r = json.loads(line) + except ValueError: + continue # a line cut by a kill; its item stays unanswered + if r.get("event") == "stopped": + discard.update(r.get("discard") or []) + elif "idx" in r: + rows.append(r) + return rows, discard + + +def q_state(results, arm, h54a=H54A): + """{idx: {"ok": True/False/None, "errors": n, "src": where}} over H54a's imported items and + every H54q hold. The first answer counts; a second error is a wrong answer (Q1 item 5).""" + a = Q_ARMS[arm] + st = {} + if a.get("h54a"): + for r in jload(os.path.join(h54a, a["h54a"], "quality.json"))["rows"]: + if r["idx"] < a["first"]: + st[r["idx"]] = {"ok": bool(r["ok"]), "errors": 0, "src": "H54a/" + a["h54a"]} + for h in holds(results): + rows, discard = q_rows(os.path.join(results, f"hold{h}", arm, "items.jsonl")) + for r in sorted(rows, key=lambda x: x.get("seq", 0)): + if r.get("seq") in discard or not (a["first"] <= r["idx"] < Q_N): + continue + s = st.setdefault(r["idx"], {"ok": None, "errors": 0, "src": None}) + if s["ok"] is not None: + continue + if r.get("error"): + s["errors"] += 1 + if s["errors"] >= 2: + s.update(ok=False, src=f"hold{h}", final_error=True) + else: + s.update(ok=bool(r["ok"]), src=f"hold{h}") + return st + + +def q_todo(results, arm, h54a=H54A): + """[(idx, errors so far)] still to ask: the items that errored once first, then the items not + asked yet, each in item order.""" + st = q_state(results, arm, h54a) + left = [(i, st.get(i, {}).get("errors", 0)) for i in range(Q_ARMS[arm]["first"], Q_N) + if st.get(i, {}).get("ok") is None] + return [x for x in left if x[1]] + [x for x in left if not x[1]] + + +def q_lost(results, arm): + """The number of holds in which ARM was not deliverable or failed.""" + n = 0 + for h in holds(results): + try: + if jload(os.path.join(results, f"hold{h}", arm, "status.json")).get("status") in LOST: + n += 1 + except (OSError, ValueError): + pass + return n + + +def q_plan(results, h54a=H54A): + """The arms the next hold runs, in Q1's order: every arm with items left, except an arm lost in + two holds (R3 carries a stopped arm once) and the probe-first Vi once it is lost.""" + return [a for a in Q_ORDER if q_todo(results, a, h54a) + and q_lost(results, a) < (1 if Q_ARMS[a].get("probe") else 2)] + + +def ready(url, model, timeout=300): + """The readiness check: the model list and a 16-token chat.""" + try: + with urllib.request.urlopen(url.rstrip("/") + "/models", timeout=10) as r: + r.read() + body = {"model": model, "max_tokens": 16, "messages": [{"role": "user", "content": "Say hello."}]} + req = urllib.request.Request(url.rstrip("/") + "/chat/completions", data=json.dumps(body).encode(), + headers={"Content-Type": "application/json"}) + with urllib.request.urlopen(req, timeout=timeout) as r: + r.read() + return True + except Exception: # noqa: BLE001 - any failure reads as not answering + return False + + +def q_ask(todo, out, stop, ask, probe, workers=16, stall_s=600, streak_max=16, probe_s=300, clock=time.time, + tick=0.5): + """Ask items `workers` in flight and append one JSON row per finished request to `out`. + + todo: [(item, errors so far)]. ask(item, timeout) returns (reply, usage) or raises; + probe(timeout) returns True when the server answers. No request starts at or after epoch `stop`; a request + still running then is cut and leaves no row, so its item stays unanswered. An item's first + error queues it once more at the back (Q1 item 5). The server is probed after `streak_max` + errors in a row, after `stall_s` s with requests in flight and none finishing, and when the + queue ends on an error, each probe given at most `probe_s` s and ending 5 s before `stop` (none + starts with less than 30 s left). A failed probe stops the arm: a `stopped` event discards the + run of errors that ended there, whose items stay unanswered.""" + def probe_by_stop(): + t = min(probe_s, stop - clock() - 5) + return None if t < 30 else probe(t) + + pending = collections.deque(todo) + lock = threading.Lock() + st = {"seq": 0, "inflight": 0, "last": clock(), "streak": [], "closed": False, "answered": 0, "errors": 0} + t_start = clock() + f = open(out, "a") + + def write(row): + f.write(json.dumps(row) + "\n") + f.flush() + + def work(it, prior): + t0 = clock() + try: + reply, usage = ask(it, max(1.0, min(1800.0, stop - t0))) + err = None + except Exception as e: # noqa: BLE001 - recorded per item + reply, usage, err = "", {}, f"{type(e).__name__}: {e}" + t1 = clock() + with lock: + st["inflight"] -= 1 + if st["closed"]: + return + st["seq"] += 1 + write({"seq": st["seq"], "idx": it["idx"], "ok": err is None and grade(reply, it["answer"]), + "gold": gold(it["answer"]), "reply": reply, "usage": usage, "error": err, "prior_errors": prior, + "t0": round(t0, 3), "t1": round(t1, 3), "s": round(t1 - t0, 2)}) + st["last"] = t1 + if err is None: + st["streak"] = [] + st["answered"] += 1 + else: + st["streak"].append(st["seq"]) + st["errors"] += 1 + if prior == 0: + pending.append((it, 1)) + + stopped, end = None, "queue empty" + while True: + now = clock() + with lock: + if now >= stop: + end = "item stop" + break + if not pending and st["inflight"] == 0: + break + while pending and st["inflight"] < workers: + it, prior = pending.popleft() + st["inflight"] += 1 + threading.Thread(target=work, args=(it, prior), daemon=True).start() + streak = len(st["streak"]) + stalled = st["inflight"] > 0 and now - st["last"] >= stall_s + if streak >= streak_max or stalled: + if probe_by_stop() is False: + stopped = f"{streak} errors in a row" if streak >= streak_max else f"no request finished in {stall_s} s" + break + with lock: + st["streak"] = [] + st["last"] = clock() + time.sleep(tick) + with lock: + trailing = bool(st["streak"]) + if stopped is None and end == "queue empty" and trailing and probe_by_stop() is False: + stopped = "the queue ended on an error" + with lock: + st["closed"] = True + if stopped: + write({"event": "stopped", "reason": f"{STOPPED}: {stopped}", "discard": st["streak"], "at": round(clock(), 3)}) + summary = {"todo": len(todo), "answered": st["answered"], "errors": st["errors"], + "discarded": len(st["streak"]) if stopped else 0, "cut": st["inflight"], + "not_asked": len({it["idx"] for it, _ in pending}), + "stopped": f"{STOPPED}: {stopped}" if stopped else None, "end": "stopped" if stopped else end, + "t0": round(t_start, 3), "t1": round(clock(), 3)} + f.close() + return summary + + +def q_items(url, model, results, arm, out, stop, h54a=H54A): + """One arm's turn in a hold: ask its q_todo items; write OUT/items.jsonl and OUT/q.json.""" + with open(os.path.join(HERE, "gsm8k-test-0-199.jsonl")) as f: + items = {r["idx"]: r for r in (json.loads(x) for x in f if x.strip())} + todo = [(items[i], e) for i, e in q_todo(results, arm, h54a)] + s = q_ask(todo, os.path.join(out, "items.jsonl"), stop, + ask=lambda it, t: chat(url, model, it["question"], timeout=t), probe=lambda t: ready(url, model, t)) + s["first_items"] = [it["idx"] for it, _ in todo[:20]] + s["status"] = "failed" if s["stopped"] else "measured" + s["reason"] = s["stopped"] or (f"{s['answered']} answered, {s['errors']} errored, {s['cut']} cut, " + f"{s['not_asked']} not asked ({s['end']})") + with open(os.path.join(out, "q.json"), "w") as f: + json.dump(s, f, indent=1) + return s + + +def paired_boot(d, n=10000, seed=1): + """95% percentile bootstrap CI of mean(d), resampling items, with H54a's percentile indices.""" + rng = random.Random(seed) + m = len(d) + means = sorted(sum(rng.choices(d, k=m)) / m for _ in range(n)) + return means[int(0.025 * n)], means[int(0.975 * n) - 1] + + +def score_q(results, h54a=H54A, n=10000, seed=1): + """The H54q row's two tests (OVQ - R, OVB - OVQ) and verdict, plus descriptive differences.""" + rep = {"arms": {}, "tests": {}, "descriptive": {}} + full = {} + for arm in Q_ORDER: + st = q_state(results, arm, h54a) + ans = {i: s["ok"] for i, s in st.items() if s["ok"] is not None} + rep["arms"][arm] = {"answered": len(ans), "correct": sum(ans.values()), + "accuracy": sum(ans.values()) / len(ans) if ans else None, + "not_answered": Q_N - len(ans), + "from_h54a": sum(1 for s in st.values() if str(s.get("src")).startswith("H54a")), + "wrong_by_error": sum(1 for s in st.values() if s.get("final_error")), + "lost_holds": q_lost(results, arm)} + if len(ans) == Q_N: + full[arm] = [int(ans[i]) for i in range(Q_N)] + + def test(a, b): + if a not in full or b not in full: + return {"missing": f"{' and '.join(x for x in (a, b) if x not in full)} did not answer all {Q_N} items"} + d = [x - y for x, y in zip(full[a], full[b])] + lo, hi = paired_boot(d, n, seed) + return {"n": Q_N, "diff": sum(d) / Q_N, "ci95": [lo, hi], + "a_only": sum(1 for x in d if x > 0), "b_only": sum(1 for x in d if x < 0)} + t1, t2 = test("OVQ", "R"), test("OVB", "OVQ") + rep["tests"] = {"OVQ-R": t1, "OVB-OVQ": t2} + for a, b in (("Lg", "R"), ("OVB", "R"), ("Vi", "R")): + rep["descriptive"][f"{a}-{b}"] = test(a, b) + ref = [] + if "ci95" in t1 and t1["ci95"][1] < -0.05: + ref.append("CI upper bound of OVQ - R < -0.05") + if "ci95" in t2 and t2["ci95"][0] > 0: + ref.append("CI lower bound of OVB - OVQ > 0") + if ref: + rep.update(verdict="REFUTED", reason="; ".join(ref)) + elif "ci95" in t1 and "ci95" in t2 and t1["ci95"][0] >= -0.05 and t2["ci95"][1] < 0: + rep.update(verdict="SUPPORTED", reason="CI lower bound of OVQ - R >= -0.05 and CI upper bound of OVB - OVQ < 0") + else: + missing = [k for k, t in rep["tests"].items() if "missing" in t] + rep.update(verdict="INCONCLUSIVE", + reason=("tests missing: " + ", ".join(missing)) if missing + else "neither the SUPPORTED bounds nor a REFUTED bound hold") + return rep + + def main(argv): if not argv: raise SystemExit(__doc__) @@ -546,6 +850,27 @@ def main(argv): print(" ".join(carry(a[0], int(a[1]), a[2:]))) elif cmd == "score-a": print(json.dumps(score_a(a[0]), indent=1)) + elif cmd == "q-base": + print(Q_ARMS[a[0]]["base"]) + elif cmd == "q-render": + q_render(a[0], a[1]) + elif cmd == "q-todo": + print(" ".join(f"{i}" + ("*" if e else "") for i, e in q_todo(a[0], a[1]))) + elif cmd == "q-items": + s = q_items(a[0], a[1], a[2], a[3], a[4], float(a[5])) + print(f"q-items {a[3]}: {s['reason']}") + return 3 if s["stopped"] else 0 + elif cmd == "q-smoke": + with open(os.path.join(HERE, "q-smoke.jsonl")) as f: + rows = [json.loads(x) for x in f if x.strip()] + s = q_ask([(r, 0) for r in rows], os.path.join(a[2], "smoke-items.jsonl"), float(a[3]), + ask=lambda it, t: chat(a[0], a[1], it["question"], timeout=t), probe=lambda t: ready(a[0], a[1], t)) + print(f"q-smoke: {json.dumps(s)}") + return 3 if s["stopped"] else 0 + elif cmd == "q-plan": + print(" ".join(q_plan(a[0]))) + elif cmd == "score-q": + print(json.dumps(score_q(a[0]), indent=1)) else: raise SystemExit(f"h54.py: unknown command {cmd}") return 0 diff --git a/bench/h54/hold.sh b/bench/h54/hold.sh index d9d08291..e9894ec3 100755 --- a/bench/h54/hold.sh +++ b/bench/h54/hold.sh @@ -15,6 +15,8 @@ HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) REPO=$(cd "${HERE}/../.." && pwd) export REPO G case ${ROW} in smoke*) R=${G}/smoke/${ROW}/hold${HOLD} ;; *) R=${G}/results/H54${ROW}/hold${HOLD} ;; esac +# H54q's arms and its wiring smoke run arm.sh's item mode (amendment Q2). +case ${ROW} in q | smokeq) export H54Q=1 ;; esac mkdir -p "${R}" export PATH=${G}/h54a/bin:${G}/bin:${PATH} EAT=${G}/h54a/eat-src export CLUSTER=k3dgpu-g-h54${ROW}-${HOLD} KUBECONFIG=${R}/kubeconfig diff --git a/bench/h54/q-smoke.jsonl b/bench/h54/q-smoke.jsonl new file mode 100644 index 00000000..047b8938 --- /dev/null +++ b/bench/h54/q-smoke.jsonl @@ -0,0 +1,2 @@ +{"idx": 1000, "question": "A box holds 3 red pens and 4 blue pens. How many pens are in the box?", "answer": "3 + 4 = 7\n#### 7"} +{"idx": 1001, "question": "Sam has 10 apples and gives away 4. How many apples does Sam have left?", "answer": "10 - 4 = 6\n#### 6"} diff --git a/bench/h54/qcampaign.sh b/bench/h54/qcampaign.sh new file mode 100755 index 00000000..a9dde40e --- /dev/null +++ b/bench/h54/qcampaign.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# qcampaign.sh [FIRST_HOLD]: H54q's holds from FIRST_HOLD (default 1) to 3 (amendment Q1's cap, +# runner amendment Q2). Before each hold, `h54.py q-plan` reads the records so far and names the +# arms with items left, in Q1's order R, OVQ, OVB, Lg, Vi; a hold runs them with hold.sh q. Items +# an arm has not answered stay at the front of its next turn. Stops early when no arm is left. +# Then not-run.txt (arms with items left after the cap) and score-q.json. +# shellcheck disable=SC2046 # the arm list splits into words on purpose +set -u +G=${HOME}/k3dgpu-gemma +H=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +R=${G}/results/H54q +mkdir -p "${R}" +for h in $(seq "${1:-1}" 3); do + arms=$(python3 "${H}/h54.py" q-plan "${R}") + if [ -z "${arms}" ]; then + echo "== H54q: no arm has items left before hold ${h} $(date -u +%FT%TZ)" | tee -a "${G}/results/campaign.log" + break + fi + echo "${arms}" > "${R}/plan-hold${h}.txt" + bash "${H}/hold.sh" q "${h}" ${arms} +done +echo "arms with items left after the cap: $(python3 "${H}/h54.py" q-plan "${R}")" > "${R}/not-run.txt" +for a in R OVQ OVB Lg Vi; do echo "${a}: $(python3 "${H}/h54.py" q-todo "${R}" "${a}" | wc -w) items not answered" >> "${R}/not-run.txt"; done +python3 "${H}/h54.py" score-q "${R}" > "${R}/score-q.json" 2>&1 +echo "== H54q campaign done $(date -u +%FT%TZ)" | tee -a "${G}/results/campaign.log" diff --git a/bench/h54/reproduce.sh b/bench/h54/reproduce.sh index 9383a3bb..41d84f47 100644 --- a/bench/h54/reproduce.sh +++ b/bench/h54/reproduce.sh @@ -29,6 +29,14 @@ done echo "weights sha256:" cat "${G}"/weights*.sha256 2>/dev/null | sed 's/^/ /' echo "data: gsm8k-test-0-199.jsonl $(sha256sum "${HERE}/gsm8k-test-0-199.jsonl" | cut -c1-16) (openai/gsm8k main test @ 740312ad); ttft-prompt.txt $(sha256sum "${HERE}/ttft-prompt.txt" | cut -c1-16)" +if [ -n "${H54Q:-}" ]; then + echo "H54q manifests (h54.py q-render, amendment Q2), sha256:" + t=$(mktemp) + for a in R OVQ OVB Lg Vi; do + python3 "${HERE}/h54.py" q-render "${a}" "${t}" && echo " ${a}: $(sha256sum "${t}" | cut -c1-16)" + done + rm -f "${t}" +fi echo "engine flags (Lg, Lg-mtp, Sg as rendered; B1g/Vq/Vi/A1o/A1g/B3g in each arm's deploy.yaml):" python3 - "${HERE}" <<'EOF' import sys diff --git a/bench/h54/test_h54.py b/bench/h54/test_h54.py index 8acaacce..497bcd2f 100644 --- a/bench/h54/test_h54.py +++ b/bench/h54/test_h54.py @@ -1,7 +1,9 @@ #!/usr/bin/env python3 """Tests for bench/h54 (EXPERIMENTS.md "H54 amendment R1"): the arm renders and their flags, the GSM8K grader, the void rule, the tensor-table byte counts, the bootstrap, the chosen-arm source -rule and the H54a verdicts. No cluster, no network.""" +rule and the H54a verdicts; and H54q's runner (amendments Q1 and Q2): its renders, item records, +hold plan, item client (against fakes and a local stub server) and scorer. No cluster, no network +beyond 127.0.0.1.""" import hashlib import json import os @@ -363,5 +365,362 @@ def test_adds_volume_only_for_campaign_create(self): self.assertEqual(out.split(), ["cluster", "list"]) +# ---- H54q (amendments Q1 and Q2) ---------------------------------------------------------------- + +def q_row(seq, idx, ok=True, error=None, prior=0): + return {"seq": seq, "idx": idx, "ok": ok and error is None, "gold": "1", "reply": "", "usage": {}, + "error": error, "prior_errors": prior, "t0": 0, "t1": 1, "s": 1} + + +def q_write(root, hold, arm, rows, status="measured", reason=""): + d = os.path.join(root, f"hold{hold}", arm) + os.makedirs(d, exist_ok=True) + with open(os.path.join(d, "items.jsonl"), "a") as f: + for r in rows: + f.write(json.dumps(r) + "\n") + with open(os.path.join(d, "status.json"), "w") as f: + json.dump({"status": status, "reason": reason}, f) + return d + + +def fake_h54a(root, correct=None): + """H54a records for the three imported arms: items 0-49, all correct unless listed wrong.""" + for arm in ("OVQ", "OVB", "Lg"): + d = os.path.join(root, h54.Q_ARMS[arm]["h54a"]) + os.makedirs(d, exist_ok=True) + wrong = (correct or {}).get(arm, set()) + with open(os.path.join(d, "quality.json"), "w") as f: + json.dump({"rows": [{"idx": i, "ok": i not in wrong, "error": None} for i in range(50)]}, f) + + +def q_full(root, arm, wrong=(), hold=1): + first = h54.Q_ARMS[arm]["first"] + q_write(root, hold, arm, [q_row(n, i, ok=i not in wrong) for n, i in enumerate(range(first, h54.Q_N), 1)]) + + +class QRecords(unittest.TestCase): + def test_h54a_items_are_score_a_records(self): + # Q1 item 2: OVQ = A1g hold 3 (49/50), OVB = A1o hold 3 (49/50), Lg = Lg hold 2 (50/50). + with tempfile.TemporaryDirectory() as d: + arms = h54.score_q(d)["arms"] + self.assertEqual({a: (arms[a]["answered"], arms[a]["correct"], arms[a]["from_h54a"]) for a in ("OVQ", "OVB", "Lg")}, + {"OVQ": (50, 49, 50), "OVB": (50, 49, 50), "Lg": (50, 50, 50)}) + self.assertEqual(arms["R"]["answered"], 0) + self.assertEqual(arms["Vi"]["answered"], 0) + + @unittest.skipUnless(HAVE_YAML, "PyYAML") + def test_renders_are_h54as(self): + import yaml + with tempfile.TemporaryDirectory() as d: + for arm, rec in (("OVQ", "hold3/A1g"), ("OVB", "hold3/A1o"), ("Lg", "hold2/Lg")): + out = os.path.join(d, arm + ".yaml") + h54.q_render(arm, out) + with open(out, "rb") as a, open(os.path.join(h54.H54A, rec, "deploy.yaml"), "rb") as b: + self.assertEqual(hashlib.sha256(a.read()).hexdigest(), hashlib.sha256(b.read()).hexdigest()) + with open(os.path.join(h54.H54A, "hold2/B1g/deploy.yaml")) as f: + b1g = [x for x in yaml.safe_load_all(f) if x] + for arm, base in (("R", "Vq"), ("Vi", "Vi")): + out = os.path.join(d, arm + ".yaml") + h54.q_render(arm, out) + with open(out) as f: + got = [x for x in yaml.safe_load_all(f) if x] + self.assertEqual(got, h54.derive_vllm(base, b1g)) + c = [x for x in got if x["kind"] == "Deployment"][0]["spec"]["template"]["spec"]["containers"][0] + self.assertEqual(c["image"], h54.IMG["vllm032"]) + model = c["args"][c["args"].index("--model") + 1] + self.assertEqual(model, f"/data/{h54.AUTOROUND if arm == 'Vi' else h54.QAT}") + self.assertNotIn("--speculative-config", c["args"]) + + def test_bases(self): + self.assertEqual({a: h54.Q_ARMS[a]["base"] for a in h54.Q_ORDER}, + {"R": "Vq", "OVQ": "A1g", "OVB": "A1o", "Lg": "Lg", "Vi": "Vi"}) + self.assertEqual(h54.Q_ORDER, ["R", "OVQ", "OVB", "Lg", "Vi"]) + for a in h54.Q_ORDER: + self.assertIn(h54.Q_ARMS[a]["base"], h54.ARMS) + self.assertEqual(h54.ARMS["Vq"]["boot"], 1500) # R is a bf16 vLLM load + + +class QState(unittest.TestCase): + def test_retry_first_second_error_wrong_and_discard(self): + with tempfile.TemporaryDirectory() as r, tempfile.TemporaryDirectory() as a: + fake_h54a(a) + q_write(r, 1, "OVQ", [q_row(1, 50), q_row(2, 51, error="HTTPError"), q_row(3, 52, ok=False), + q_row(4, 53, error="x"), q_row(5, 53, error="x", prior=1), + q_row(6, 54, error="x"), {"event": "stopped", "discard": [6], "reason": "r"}]) + st = h54.q_state(r, "OVQ", a) + self.assertTrue(st[50]["ok"]) + self.assertIsNone(st[51]["ok"]) + self.assertEqual(st[51]["errors"], 1) + self.assertFalse(st[52]["ok"]) + self.assertFalse(st[53]["ok"]) + self.assertTrue(st[53]["final_error"]) + self.assertNotIn(54, st) # discarded by the stopped event + todo = h54.q_todo(r, "OVQ", a) + self.assertEqual(todo[0], (51, 1)) # errored once: front of the next turn + self.assertEqual(todo[1], (54, 0)) + self.assertEqual(len(todo), 150 - 3) + # hold 2 answers 51 on its second ask: the answer counts, not the error + q_write(r, 2, "OVQ", [q_row(1, 51, ok=True, prior=1)]) + self.assertTrue(h54.q_state(r, "OVQ", a)[51]["ok"]) + + def test_first_answer_wins_and_range(self): + with tempfile.TemporaryDirectory() as r, tempfile.TemporaryDirectory() as a: + fake_h54a(a) + q_write(r, 1, "OVQ", [q_row(1, 60, ok=False), q_row(2, 7, ok=False)]) # 7 is H54a's + q_write(r, 2, "OVQ", [q_row(1, 60, ok=True)]) + st = h54.q_state(r, "OVQ", a) + self.assertFalse(st[60]["ok"]) + self.assertTrue(st[7]["ok"]) + self.assertEqual(st[7]["src"], "H54a/hold3/A1g") + self.assertEqual(h54.q_todo(r, "R", a)[0], (0, 0)) # R answers 0-199 + + def test_cut_line_is_unanswered(self): + with tempfile.TemporaryDirectory() as r, tempfile.TemporaryDirectory() as a: + fake_h54a(a) + d = q_write(r, 1, "Lg", [q_row(1, 50)]) + with open(os.path.join(d, "items.jsonl"), "a") as f: + f.write('{"seq": 2, "idx": 51, "ok": tr') # killed mid-write + self.assertEqual(h54.q_todo(r, "Lg", a)[0], (51, 0)) + + +class QPlan(unittest.TestCase): + def test_order_done_and_lost(self): + with tempfile.TemporaryDirectory() as r, tempfile.TemporaryDirectory() as a: + fake_h54a(a) + self.assertEqual(h54.q_plan(r, a), ["R", "OVQ", "OVB", "Lg", "Vi"]) + q_full(r, "R") + q_write(r, 1, "OVQ", [q_row(1, 50)]) + self.assertEqual(h54.q_plan(r, a), ["OVQ", "OVB", "Lg", "Vi"]) + # not deliverable or stopped once: carried; twice: dropped (R3 carries once) + q_write(r, 1, "OVB", [], status="failed", reason="server stopped answering: x") + self.assertIn("OVB", h54.q_plan(r, a)) + q_write(r, 2, "OVB", [], status="not_deliverable") + self.assertNotIn("OVB", h54.q_plan(r, a)) + # not reached costs nothing + q_write(r, 1, "Lg", [], status="not_reached") + q_write(r, 2, "Lg", [], status="not_reached") + self.assertIn("Lg", h54.q_plan(r, a)) + # the probe-first Vi gets one boot attempt + q_write(r, 2, "Vi", [], status="not_deliverable") + self.assertEqual(h54.q_plan(r, a), ["OVQ", "Lg"]) + + def test_cli(self): + with tempfile.TemporaryDirectory() as r: + run = lambda *a: subprocess.run([sys.executable, os.path.join(HERE, "h54.py"), *a], capture_output=True, text=True) # noqa: E731 + self.assertEqual(run("q-plan", r).stdout.split(), ["R", "OVQ", "OVB", "Lg", "Vi"]) + self.assertEqual(run("q-base", "OVB").stdout.strip(), "A1o") + self.assertEqual(run("q-todo", r, "OVQ").stdout.split()[:2], ["50", "51"]) + + +def item(i): + return {"idx": i, "question": f"q{i}", "answer": f"#### {i}"} + + +class QAsk(unittest.TestCase): + def ask_with(self, todo, ask, probe=lambda t: True, stop=None, **kw): + import time + with tempfile.TemporaryDirectory() as d: + out = os.path.join(d, "items.jsonl") + s = h54.q_ask(todo, out, stop or time.time() + 60, ask, probe, tick=0.01, **kw) + rows, discard = h54.q_rows(out) + return s, rows, discard + + def test_answers_and_one_retry(self): + seen = {} + + def ask(it, t): + seen[it["idx"]] = seen.get(it["idx"], 0) + 1 + if it["idx"] == 3 and seen[3] == 1: + raise OSError("reset") + if it["idx"] == 4: + raise OSError("always") + return f"Answer: {it['idx'] if it['idx'] != 2 else 99}", {"completion_tokens": 5} + s, rows, _ = self.ask_with([(item(i), 0) for i in range(5)] + [(item(5), 1)], ask, workers=2) + self.assertEqual(seen, {0: 1, 1: 1, 2: 1, 3: 2, 4: 2, 5: 1}) # an error is asked once more + self.assertIsNone(s["stopped"]) + self.assertEqual((s["answered"], s["errors"], s["cut"], s["not_asked"]), (5, 3, 0, 0)) + by = {} + for r in rows: + by.setdefault(r["idx"], []).append(r) + self.assertEqual([r["ok"] for r in by[2]], [False]) + self.assertEqual([bool(r["error"]) for r in by[3]], [True, False]) + self.assertEqual([r["prior_errors"] for r in by[4]], [0, 1]) + self.assertEqual(len(by[5]), 1) # errored before: its retry is this ask + + def test_streak_stops_and_discards(self): + def ask(it, t): + raise ConnectionRefusedError("down") + probes = [] + + def probe(t): + probes.append(t) + return False + s, rows, discard = self.ask_with([(item(i), 0) for i in range(40)], ask, probe=probe, workers=4, streak_max=4) + self.assertTrue(s["stopped"].startswith(h54.STOPPED)) + self.assertEqual(len(probes), 1) + self.assertTrue(30 <= probes[0] <= 300) # bounded by probe_s and by the stop + self.assertEqual(discard, {r["seq"] for r in rows}) # every error of the run is discarded + self.assertGreater(s["not_asked"], 0) + + def test_no_probe_near_the_item_stop(self): + import time + + def ask(it, t): + raise ConnectionRefusedError("down") + probes = [] + s, rows, discard = self.ask_with([(item(i), 1) for i in range(8)], ask, probe=lambda t: probes.append(t), + stop=time.time() + 20, workers=4, streak_max=4) + self.assertEqual(probes, []) # under 30 s before the stop: no probe + self.assertIsNone(s["stopped"]) + self.assertEqual((s["errors"], discard), (8, set())) + + def test_streak_with_answering_server_continues(self): + def ask(it, t): + if it["idx"] < 8: + raise OSError("bad request") + return f"Answer: {it['idx']}", {} + s, rows, discard = self.ask_with([(item(i), 1) for i in range(8)] + [(item(i), 0) for i in range(8, 12)], + ask, workers=4, streak_max=4) + self.assertIsNone(s["stopped"]) + self.assertEqual(discard, set()) + self.assertEqual((s["answered"], s["errors"]), (4, 8)) + + def test_trailing_error_probe(self): + def ask(it, t): + if it["idx"] == 2: + raise OSError("down") + return "Answer: 1", {} + s, _, discard = self.ask_with([(item(i), 0) for i in range(3)], ask, probe=lambda t: False, workers=1) + self.assertIn("ended on an error", s["stopped"]) + self.assertEqual(len(discard), 2) # the item's ask and its retry + + def test_item_stop_cuts(self): + import time + + def ask(it, t): + time.sleep(min(t, 0.5) if it["idx"] else 0.0) + return f"Answer: {it['idx']}", {} + s, rows, _ = self.ask_with([(item(i), 0) for i in range(10)], ask, stop=time.time() + 0.3, workers=2) + self.assertEqual(s["end"], "item stop") + self.assertEqual(s["cut"], 2) + self.assertEqual(s["not_asked"], 7) + self.assertEqual([r["idx"] for r in rows], [0]) + + def test_stall_probe(self): + import time + + def ask(it, t): + time.sleep(2) + return "x", {} + s, rows, _ = self.ask_with([(item(i), 0) for i in range(3)], ask, probe=lambda t: False, workers=3, stall_s=0.2) + self.assertIn("no request finished", s["stopped"]) + self.assertEqual(rows, []) + + def test_against_a_stub_server(self): + import http.server + import threading + + class Stub(http.server.BaseHTTPRequestHandler): + def log_message(self, *a): + pass + + def reply(self, body): + b = json.dumps(body).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(b))) + self.end_headers() + self.wfile.write(b) + + def do_GET(self): + self.reply({"data": [{"id": "gemma"}]}) + + def do_POST(self): + req = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) + q = req["messages"][0]["content"] + assert req.get("chat_template_kwargs") == {"enable_thinking": False} or req["max_tokens"] == 16 + self.reply({"choices": [{"message": {"content": "Answer: 7" if "pens" in q else "Answer: 5"}}], + "usage": {"completion_tokens": 3}}) + srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Stub) + threading.Thread(target=srv.serve_forever, daemon=True).start() + url = f"http://127.0.0.1:{srv.server_address[1]}/v1" + try: + with tempfile.TemporaryDirectory() as d: + self.assertTrue(h54.ready(url, "gemma")) + p = subprocess.run([sys.executable, os.path.join(HERE, "h54.py"), "q-smoke", url, "gemma", d, + str(__import__("time").time() + 60)], capture_output=True, text=True) + self.assertEqual(p.returncode, 0, p.stdout + p.stderr) + rows, _ = h54.q_rows(os.path.join(d, "smoke-items.jsonl")) + self.assertEqual(sorted((r["idx"], r["ok"]) for r in rows), [(1000, True), (1001, False)]) + res, out = os.path.join(d, "res"), os.path.join(d, "res", "hold1", "Lg") + os.makedirs(out) + p = subprocess.run([sys.executable, os.path.join(HERE, "h54.py"), "q-items", url, "gemma", res, "Lg", out, + str(__import__("time").time() + 60)], capture_output=True, text=True) + self.assertEqual(p.returncode, 0, p.stdout + p.stderr) + q = json.load(open(os.path.join(out, "q.json"))) + self.assertEqual((q["status"], q["todo"], q["answered"], q["first_items"][0]), ("measured", 150, 150, 50)) + self.assertEqual(h54.q_todo(res, "Lg"), []) + finally: + srv.shutdown() + self.assertFalse(h54.ready("http://127.0.0.1:9/v1", "gemma", timeout=1)) + + +class ScoreQ(unittest.TestCase): + def score(self, wrong, h54a_wrong=None): + with tempfile.TemporaryDirectory() as r, tempfile.TemporaryDirectory() as a: + fake_h54a(a, h54a_wrong) + for arm, w in wrong.items(): + q_full(r, arm, w) + return h54.score_q(r, a, n=2000) + + def test_supported(self): + # OVQ = R; OVB misses 20 items OVQ answers + rep = self.score({"R": set(range(50, 200, 25)), "OVQ": set(range(50, 200, 25)), "OVB": set(range(51, 71))}) + self.assertEqual(rep["verdict"], "SUPPORTED", rep) + self.assertEqual(rep["tests"]["OVQ-R"]["diff"], 0.0) + self.assertLess(rep["tests"]["OVB-OVQ"]["ci95"][1], 0) + + def test_refuted_ovq_below_r(self): + rep = self.score({"R": set(), "OVQ": set(range(60, 100)), "OVB": set(range(60, 100))}) + self.assertEqual(rep["verdict"], "REFUTED") + self.assertIn("OVQ - R", rep["reason"]) + + def test_refuted_ovb_above_ovq_with_one_test(self): + # R did not answer every item: OVQ - R is missing, and the OVB - OVQ test alone refutes + with tempfile.TemporaryDirectory() as r, tempfile.TemporaryDirectory() as a: + fake_h54a(a) + q_full(r, "OVQ", set(range(60, 90))) + q_full(r, "OVB", set()) + q_write(r, 1, "R", [q_row(1, 0)]) + rep = h54.score_q(r, a, n=2000) + self.assertIn("missing", rep["tests"]["OVQ-R"]) + self.assertEqual(rep["verdict"], "REFUTED") + self.assertIn("OVB - OVQ", rep["reason"]) + + def test_missing_is_inconclusive(self): + rep = self.score({"OVQ": set(), "OVB": set(range(60, 90))}) + self.assertEqual(rep["verdict"], "INCONCLUSIVE") + self.assertIn("OVQ-R", rep["reason"]) + self.assertEqual(rep["tests"]["OVQ-R"]["missing"], "R did not answer all 200 items") + + def test_neither(self): + rep = self.score({"R": set(), "OVQ": set(), "OVB": set()}) + self.assertEqual(rep["verdict"], "INCONCLUSIVE") + self.assertEqual(rep["tests"]["OVB-OVQ"]["ci95"], [0.0, 0.0]) + + def test_imported_items_count(self): + rep = self.score({"R": set(), "OVQ": set(), "OVB": set()}, h54a_wrong={"OVQ": {3}}) + self.assertEqual(rep["arms"]["OVQ"]["correct"], 199) + self.assertEqual(rep["tests"]["OVQ-R"]["b_only"], 1) + + def test_paired_boot(self): + d = [1] * 10 + [0] * 180 + [-1] * 10 + self.assertEqual(h54.paired_boot(d, 2000), h54.paired_boot(d, 2000)) + lo, hi = h54.paired_boot(d, 2000) + self.assertLess(lo, 0) + self.assertGreater(hi, 0) + self.assertEqual(h54.paired_boot([0.5] * 7, 100), (0.5, 0.5)) + + if __name__ == "__main__": unittest.main() From 85e77267a059578d412461cf69ebf5680170802b Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Thu, 29 Oct 2026 12:36:10 -0700 Subject: [PATCH 02/21] bench(h54q): qcheck.sh, the weight check and the serving front-door record around the holds Neither step is part of H54q's protocol or score. verify hashes every weight file the arms load against the pre-registration's sha256 list and OVQ's manifest, under the bench lock. frontdoor records that both SGLang serving pairs still answer through Dynamo :8100, with each pair's request counter rising. --- bench/h54/qcheck.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 bench/h54/qcheck.sh diff --git a/bench/h54/qcheck.sh b/bench/h54/qcheck.sh new file mode 100644 index 00000000..8196f5ad --- /dev/null +++ b/bench/h54/qcheck.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# qcheck.sh STEP: H54q's checks on crescent around its holds. Neither step is part of the +# protocol or the score; both are records. +# verify under the bench lock: the sha256 of every weight file H54q's arms load, against +# ~/k3dgpu-gemma/weights*.sha256 (the H54 pre-registration's list) and OVQ's +# manifest.json; writes results/H54q/weights-verify.txt +# frontdoor L a 16-token chat through Dynamo :8100 for each serving pair, passing when it +# answers with content and that pair's SGLang num_requests_total rises (180 s each, +# E2 item 7's check); writes results/H54q/frontdoor-L.txt. H54q uses no card and +# never stops a serving pair; this records that both still serve. +set -uo pipefail +G=${HOME}/k3dgpu-gemma +M=${G}/models +O=${G}/results/H54q +mkdir -p "${O}" +case ${1:?usage: qcheck.sh verify|frontdoor LABEL} in +verify) + exec 9> "${HOME}/k3dgpu-bench.lock" + flock -w 21600 9 || { echo "lock not taken in 21600 s"; exit 4; } + f=${O}/weights-verify.txt + { + date -u +%FT%TZ + # weights.sha256 names OVB's download directory; R1 renamed it to the Hub id's path. + sed 's# OpenVINO_gemma-4-26b-a4b-it-int4-ov/# OpenVINO/gemma-4-26b-a4b-it-int4-ov/#' "${G}/weights.sha256" \ + | (cd "${M}" && sha256sum -c -) + echo "weights.sha256 rc=$?" + (cd "${M}/Intel_gemma-4-26B-A4B-it-int4-AutoRound" && sha256sum -c "${G}/weights-autoround.sha256") + echo "weights-autoround.sha256 rc=$?" + python3 - "${M}/k3dgpu/gemma-4-26b-a4b-it-qat-int4-sym-g32-ov" <<'PY' +import hashlib, json, os, sys +d = sys.argv[1] +files = json.load(open(os.path.join(d, "manifest.json")))["files"] +bad = 0 +for n, want in sorted(files.items()): + h = hashlib.sha256() + with open(os.path.join(d, n), "rb") as f: + for chunk in iter(lambda: f.read(1 << 24), b""): + h.update(chunk) + ok = h.hexdigest() == want["sha256"] and os.path.getsize(os.path.join(d, n)) == want["bytes"] + bad += not ok + print(f"{'OK ' if ok else 'BAD'} OVQ {n} {h.hexdigest()[:16]} {want['bytes']}") +print(f"OVQ {d}: {len(files)} files, {bad} bad; not in the manifest: {sorted(set(os.listdir(d)) - set(files) - {'manifest.json'})}") +sys.exit(1 if bad else 0) +PY + echo "OVQ manifest rc=$?" + for d in google_gemma-4-26B-A4B-it-qat-q4_0-unquantized google_gemma-4-26B-A4B-it-qat-q4_0-gguf \ + OpenVINO/gemma-4-26b-a4b-it-int4-ov Intel_gemma-4-26B-A4B-it-int4-AutoRound; do + echo "${d} revision $(head -1 "${M}/${d}/.pinned-revision" 2>/dev/null || echo none)" + done + date -u +%FT%TZ + } > "${f}" 2>&1 + grep -E "rc=|FAILED|BAD|files," "${f}" ;; +frontdoor) + f=${O}/frontdoor-${2:?usage: qcheck.sh frontdoor LABEL}.txt + : > "${f}" + n() { curl -s -m 8 "http://127.0.0.1:$1/metrics" | awk '/^sglang:num_requests_total/{s+=$2} END{print s+0}'; } + for P in 30000 30001; do + b=$(n "${P}") end=$(( SECONDS + 180 )) ok=0 c="" + until [ "${SECONDS}" -ge "${end}" ]; do + c=$(curl -s -m 60 http://127.0.0.1:8100/v1/chat/completions -H 'Content-Type: application/json' \ + -d '{"model":"Qwen3.8-27B","messages":[{"role":"user","content":"Say OK"}],"max_tokens":16,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}' \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["choices"][0]["message"]["content"].strip())' 2>/dev/null) + if [ -n "${c}" ] && [ "$(n "${P}")" -gt "${b}" ]; then ok=1; break; fi + sleep 5 + done + echo "$(date -u +%FT%TZ) :${P} pass=${ok} before=${b} after=$(n "${P}") content=${c:-none}" | tee -a "${f}" + done ;; +*) echo "usage: qcheck.sh verify|frontdoor LABEL"; exit 2 ;; +esac From 14eef6dea12d471d506f8f1f23033457cee592d1 Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Thu, 29 Oct 2026 13:52:00 -0700 Subject: [PATCH 03/21] eas: k3s/k3d patch overlay that drops each patch once upstream carries the fix integrations/eas/overlay.py decides, per row of integrations/eas/overlay.json and at one EAS commit, whether a k3s/k3d patch is skipped (not needed on this cluster), dropped because upstream merged it (git apply -R --check), dropped because upstream fixed it another way (a probe matches and the knob the generator writes is still there), applied (first variant whose git apply --check passes), or a STOP that names the upstream change that moved the hunk and says to regenerate. Every decision prints as `fact -> choice [source]`, with --explain giving permalinks. Decisions read the commit through a scratch index, so they do not change once the overlay is applied; --apply writes the working tree only when nothing STOPs, and a re-run is a no-op. Patches are looked up in integrations/eas/patches, then in the upstream drafts, so the drafts stay the only copy. The one new patch is the release-1.2 form of 0006, where the EnvoyProxy task is still in roles/envoy_gateway/tasks/install.yaml. tests/eas_test.sh (section overlay) runs it on synthetic git repos built from byte-identical copies of the EAS files the patches touch, at main 97f6157 and release-1.2 5c27fb3. apply-check.sh gains --overlay, which checks the recorded decisions on a real clone and prints them for branch tips and open EAS PRs #50, #52 and #61. --- .github/workflows/ci.yml | 1 + integrations/eas/overlay.json | 164 ++++++ integrations/eas/overlay.py | 469 ++++++++++++++++ ...ase-1.2-envoy-control-plane-affinity.patch | 77 +++ integrations/eat/upstream/README.md | 17 +- .../eat/upstream/checks/apply-check.sh | 30 +- tests/eas_test.sh | 437 +++++++++++++++ .../configs/defaults/global_config.yaml | 152 ++++++ .../fixtures/eas/trees/5c27fb3/docs/README.md | 61 +++ .../5c27fb3/docs/customize/node_topology.md | 267 +++++++++ .../trees/5c27fb3/docs/deploy/topologies.md | 241 ++++++++ .../roles/envoy_gateway/tasks/install.yaml | 513 ++++++++++++++++++ .../5c27fb3/roles/istio/defaults/main.yaml | 55 ++ .../5c27fb3/roles/istio/tasks/install.yaml | 122 +++++ .../5c27fb3/roles/istio/tasks/upgrade.yaml | 23 + .../kubernetes/tasks/apply_node_topology.yaml | 106 ++++ .../roles/kubernetes/tasks/install.yaml | 219 ++++++++ .../roles/nri_cpu_balloons/defaults/main.yaml | 61 +++ .../tasks/install_advanced.yaml | 396 ++++++++++++++ .../configs/defaults/global_config.yaml | 159 ++++++ .../fixtures/eas/trees/97f6157/docs/README.md | 62 +++ .../97f6157/docs/customize/node_topology.md | 267 +++++++++ .../trees/97f6157/docs/deploy/topologies.md | 240 ++++++++ .../roles/envoy_gateway/tasks/controller.yaml | 195 +++++++ .../97f6157/roles/istio/defaults/main.yaml | 50 ++ .../97f6157/roles/istio/tasks/install.yaml | 117 ++++ .../97f6157/roles/istio/tasks/upgrade.yaml | 18 + .../kubernetes/tasks/apply_node_topology.yaml | 106 ++++ .../roles/kubernetes/tasks/install.yaml | 217 ++++++++ .../roles/nri_cpu_balloons/defaults/main.yaml | 61 +++ .../tasks/install_advanced.yaml | 396 ++++++++++++++ tests/fixtures/eas/trees/LICENSE | 223 ++++++++ tests/fixtures/eas/trees/SOURCES | 35 ++ 33 files changed, 5555 insertions(+), 2 deletions(-) create mode 100644 integrations/eas/overlay.json create mode 100755 integrations/eas/overlay.py create mode 100644 integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch create mode 100755 tests/eas_test.sh create mode 100644 tests/fixtures/eas/trees/5c27fb3/configs/defaults/global_config.yaml create mode 100644 tests/fixtures/eas/trees/5c27fb3/docs/README.md create mode 100644 tests/fixtures/eas/trees/5c27fb3/docs/customize/node_topology.md create mode 100644 tests/fixtures/eas/trees/5c27fb3/docs/deploy/topologies.md create mode 100644 tests/fixtures/eas/trees/5c27fb3/roles/envoy_gateway/tasks/install.yaml create mode 100644 tests/fixtures/eas/trees/5c27fb3/roles/istio/defaults/main.yaml create mode 100644 tests/fixtures/eas/trees/5c27fb3/roles/istio/tasks/install.yaml create mode 100644 tests/fixtures/eas/trees/5c27fb3/roles/istio/tasks/upgrade.yaml create mode 100644 tests/fixtures/eas/trees/5c27fb3/roles/kubernetes/tasks/apply_node_topology.yaml create mode 100644 tests/fixtures/eas/trees/5c27fb3/roles/kubernetes/tasks/install.yaml create mode 100644 tests/fixtures/eas/trees/5c27fb3/roles/nri_cpu_balloons/defaults/main.yaml create mode 100644 tests/fixtures/eas/trees/5c27fb3/roles/nri_cpu_balloons/tasks/install_advanced.yaml create mode 100644 tests/fixtures/eas/trees/97f6157/configs/defaults/global_config.yaml create mode 100644 tests/fixtures/eas/trees/97f6157/docs/README.md create mode 100644 tests/fixtures/eas/trees/97f6157/docs/customize/node_topology.md create mode 100644 tests/fixtures/eas/trees/97f6157/docs/deploy/topologies.md create mode 100644 tests/fixtures/eas/trees/97f6157/roles/envoy_gateway/tasks/controller.yaml create mode 100644 tests/fixtures/eas/trees/97f6157/roles/istio/defaults/main.yaml create mode 100644 tests/fixtures/eas/trees/97f6157/roles/istio/tasks/install.yaml create mode 100644 tests/fixtures/eas/trees/97f6157/roles/istio/tasks/upgrade.yaml create mode 100644 tests/fixtures/eas/trees/97f6157/roles/kubernetes/tasks/apply_node_topology.yaml create mode 100644 tests/fixtures/eas/trees/97f6157/roles/kubernetes/tasks/install.yaml create mode 100644 tests/fixtures/eas/trees/97f6157/roles/nri_cpu_balloons/defaults/main.yaml create mode 100644 tests/fixtures/eas/trees/97f6157/roles/nri_cpu_balloons/tasks/install_advanced.yaml create mode 100644 tests/fixtures/eas/trees/LICENSE create mode 100644 tests/fixtures/eas/trees/SOURCES diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 00bf2c28..f8dc763a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,6 +47,7 @@ jobs: - run: tests/image_test.sh - run: tests/host_setup_test.sh - run: tests/eat_test.sh + - run: tests/eas_test.sh - run: tests/platforms_doc_test.sh - run: tests/reserved_cpus_test.sh - run: tests/experimental_test.sh diff --git a/integrations/eas/overlay.json b/integrations/eas/overlay.json new file mode 100644 index 00000000..f44b425a --- /dev/null +++ b/integrations/eas/overlay.json @@ -0,0 +1,164 @@ +{ + "schema": "k3d-gpu.eas-overlay/v1", + "checked": "2026-10-29", + "repo": "intel/enterprise-ai-solutions", + "patch_dirs": ["integrations/eas/patches", "integrations/eat/upstream/patches"], + "rules": { + "always": "every k3d-gpu cluster", + "eas_owns_nri": "only when kubernetes_cpu_policy is nri-balloons: EAS installs the nri-plugins balloons chart only then (configs/components.yaml, nri_cpu_balloons.enabled)", + "multi_node": "only on a cluster with more than one node; k3d-gpu runs one server", + "never": "documentation only; nothing at install time reads it" + }, + "rows": [ + { + "id": "0002", + "what": "istio: pass istio_platform to the Istio chart as global.platform, so istio-cni uses the k3s/k3d CNI paths", + "variants": [ + {"name": "a", "files": ["0002-eas-istio-global-platform.patch"], "for": "main and release-1.2"} + ], + "needed_when": "always", + "fixed_probe": [ + { + "glob": "roles/istio/**", + "regex": "(?m)global\\.platform\\s*=|['\"]platform['\"]\\s*:|global:\\s*\\{\\s*platform\\s*:|^\\s*global:[ \\t]*\\n\\s+platform:", + "knob": "istio_platform", + "what": "an Istio task or default sets global.platform from a variable" + } + ], + "moved": [ + { + "glob": "configs/defaults/global_config.yaml", + "regex": "(?m)^headlamp_enabled:", + "upstream": "https://github.com/intel/enterprise-ai-solutions/pull/61", + "what": "headlamp_enabled now follows velero_enabled, the context of 0002's global_config.yaml hunk" + } + ], + "upstream": { + "filed": false, + "evidence": "https://istio.io/latest/docs/ambient/install/platform-prerequisites/" + } + }, + { + "id": "0003", + "what": "nri_cpu_balloons: pin the nri-plugins balloons chart to v0.15.0", + "variants": [ + {"name": "a", "files": ["0003-eas-pin-nri-balloons-chart.patch"], "for": "main and release-1.2"} + ], + "needed_when": "eas_owns_nri", + "fixed_probe": [ + { + "glob": "roles/nri_cpu_balloons/tasks/install_advanced.yaml", + "regex": "(?m)^\\s*chart_version:", + "knob": null, + "what": "the balloons Helm task passes a chart_version" + } + ], + "moved": [], + "upstream": { + "filed": false, + "evidence": "https://github.com/containers/nri-plugins/releases/tag/v0.15.0" + } + }, + { + "id": "0004", + "what": "kubernetes: apply the workload-class node labels on multi-node BYO clusters", + "variants": [ + {"name": "a", "files": ["0004-eas-byo-node-topology-labels.patch"], "for": "main and release-1.2"} + ], + "needed_when": "multi_node", + "fixed_probe": [ + { + "glob": "roles/kubernetes/tasks/install.yaml", + "regex": "apply_node_topology\\.yaml", + "knob": null, + "what": "the role's install entry point runs apply_node_topology.yaml itself" + } + ], + "moved": [], + "upstream": { + "filed": false, + "evidence": "https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/roles/kubernetes/tasks/install.yaml#L23-L25" + } + }, + { + "id": "0005", + "what": "docs: Single-Node Dev on k3d", + "variants": [ + {"name": "a", "files": ["0005-eas-docs-single-node-k3d.patch"], "for": "main"} + ], + "needed_when": "never", + "fixed_probe": [], + "moved": [], + "upstream": { + "filed": false, + "evidence": "https://github.com/intel/enterprise-ai-solutions/blob/97f6157d6a72f7173c698c474d4e51d2c79f1fd6/docs/deploy/topologies.md#L189" + } + }, + { + "id": "0006", + "what": "envoy_gateway: select control-plane nodes by label presence (k3s stamps \"true\", kubeadm \"\")", + "variants": [ + {"name": "a", "files": ["0006-eas-envoy-control-plane-affinity.patch"], "for": "main after EAS #42 (roles/envoy_gateway/tasks/controller.yaml)"}, + {"name": "b", "files": ["0006-release-1.2-envoy-control-plane-affinity.patch"], "for": "release-1.2 (roles/envoy_gateway/tasks/install.yaml)"} + ], + "needed_when": "always", + "fixed_probe": [ + { + "glob": "roles/envoy_gateway/**", + "regex": "(?s)\\A(?!.*node-role\\.kubernetes\\.io/control-plane:[ \\t]*(?:\"\"|''))(?=.*?(?PmatchExpressions:\\s*-[ \\t]*key:[ \\t]*[\"']?node-role\\.kubernetes\\.io/control-plane[\"']?\\s+operator:[ \\t]*[\"']?Exists))", + "knob": null, + "what": "a file selects control-plane nodes with a matchExpressions Exists term and has no exact \"\" selector" + }, + { + "glob": "roles/envoy_gateway/vars/k3s.yaml", + "regex": "(?m)^_envoy_gateway_proxy_shape:", + "knob": null, + "what": "a k3s entry in the per-distribution vars layout of EAS #50" + } + ], + "moved": [ + { + "glob": "roles/envoy_gateway/tasks/controller.yaml", + "regex": "_eg_k8s_provider:", + "upstream": "https://github.com/intel/enterprise-ai-solutions/pull/50", + "what": "the EnvoyProxy provider moved into the _eg_k8s_provider var, still an exact \"\" selector (#52 carries the same change)" + } + ], + "upstream": { + "filed": false, + "evidence": "https://github.com/k3s-io/k3s/blob/843c32de489c25ccc1bde53d21c5fd7df23c706e/pkg/server/server.go#L550" + } + } + ], + "verified": [ + { + "profile": "main", + "rev": "97f6157d6a72f7173c698c474d4e51d2c79f1fd6", + "fixture": "tests/fixtures/eas/trees/97f6157", + "cases": [ + {"policy": "best-effort", "nodes": 1, + "expect": {"0002": "apply:a", "0003": "skip", "0004": "skip", "0005": "skip", "0006": "apply:a"}}, + {"policy": "nri-balloons", "nodes": 3, + "expect": {"0002": "apply:a", "0003": "apply:a", "0004": "apply:a", "0005": "skip", "0006": "apply:a"}} + ] + }, + { + "profile": "v1.2.0-rc.1", + "rev": "5c27fb39a95eecec41aa4e29a1188984e9c726f3", + "fixture": "tests/fixtures/eas/trees/5c27fb3", + "cases": [ + {"policy": "best-effort", "nodes": 1, + "expect": {"0002": "apply:a", "0003": "skip", "0004": "skip", "0005": "skip", "0006": "apply:b"}}, + {"policy": "nri-balloons", "nodes": 3, + "expect": {"0002": "apply:a", "0003": "apply:a", "0004": "apply:a", "0005": "skip", "0006": "apply:b"}} + ] + } + ], + "watch": [ + {"ref": "origin/main", "what": "EAS main tip"}, + {"ref": "origin/release-1.2", "what": "EAS release-1.2 tip"}, + {"ref": "pull/50/head", "what": "open: OpenShift platform layer, moves the EnvoyProxy provider into _eg_k8s_provider"}, + {"ref": "pull/52/head", "what": "open: OpenShift ownership labels, stacked on #50"}, + {"ref": "pull/61/head", "what": "open: Headlamp SSO, changes 0002's global_config.yaml context and 0004/0005"} + ] +} diff --git a/integrations/eas/overlay.py b/integrations/eas/overlay.py new file mode 100755 index 00000000..c4c39b47 --- /dev/null +++ b/integrations/eas/overlay.py @@ -0,0 +1,469 @@ +#!/usr/bin/env python3 +"""The k3s/k3d patch overlay for an intel/enterprise-ai-solutions (EAS) checkout. + +For each row of overlay.json it decides, at one EAS commit, and prints the +decision as `fact -> choice [source]`: + + skip the row's needed_when rule is false for these facts + drop-merged a variant reverse-applies (git apply -R --check): upstream has it + drop-fixed a fixed_probe regex matches a file its glob selects, and the knob + the generator writes appears in that file (no knob: the match + alone). A match whose knob is gone is a STOP: upstream fixed it + under another name, and the generator's knob would do nothing. + apply the first variant, in order, that applies (git apply --check) + STOP none of the above: the patch has to be regenerated. The line + names the upstream change that moved the hunk: a `moved` hint of + the row when one matches, else the last commit on the patch's files. + +Decisions read the commit (--rev, default HEAD) through a scratch index, never +the working tree, so they do not change once the overlay is applied. --apply +patches the working tree of the checkout, only when --rev is HEAD and no row +STOPs: every chosen patch the working tree lacks, in one `git apply`. A patch +it already carries (a re-run) is left as it is; a patch's files in any other +state STOP the run before anything is written. + +Patch files are looked up in the overlay.json patch_dirs in order +(integrations/eas/patches, then the upstream drafts in +integrations/eat/upstream/patches), so a draft is never copied. + +Usage: + overlay.py --eas DIR --policy POLICY --nodes N [--rev REV] [--apply] + [--json FILE|-] [--explain] + overlay.py --eas DIR --check-verified every overlay.json `verified` case at its sha + overlay.py --eas DIR --watch decisions at every `watch` ref present in DIR + +POLICY is the kubernetes_cpu_policy the generator chose (nri-balloons, +kubelet-static or best-effort); N is the cluster's node count. Exit 0 when +nothing STOPs, 3 on a STOP (nothing applied), 1 when --check-verified finds a +moved decision, 2 on bad input. +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(os.path.dirname(HERE)) +POLICIES = ("nri-balloons", "kubelet-static", "best-effort") +RULES = { + "always": lambda f: True, + "eas_owns_nri": lambda f: f["policy"] == "nri-balloons", + "multi_node": lambda f: f["nodes"] > 1, + "never": lambda f: False, +} +# The k3d-gpu cluster: one server, its own balloons policy kept (--watch). +K3D_GPU_FACTS = {"policy": "best-effort", "nodes": 1} + + +class Bad(Exception): + """Input the overlay cannot work with (exit 2).""" + + +def glob_re(glob): + """A path glob as a regex: ** spans directories, * and ? stay within one.""" + out, i = "", 0 + while i < len(glob): + if glob.startswith("**/", i): + out, i = out + "(?:.*/)?", i + 3 + elif glob.startswith("**", i): + out, i = out + ".*", i + 2 + elif glob[i] == "*": + out, i = out + "[^/]*", i + 1 + elif glob[i] == "?": + out, i = out + "[^/]", i + 1 + else: + out, i = out + re.escape(glob[i]), i + 1 + return re.compile(out + r"\Z") + + +def first_hunk(path): + """The first hunk of a patch: (file, {"old"|"new": (stated start, lines)}, leading context).""" + target, start, sides, ctx, changed = None, None, {"old": [], "new": []}, 0, False + with open(path, encoding="utf-8") as f: + for line in f: + if start is None: + if line.startswith("+++ "): + target = line[4:].rstrip("\n") + target = target[2:] if target.startswith("b/") else target + m = re.match(r"@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))?", line) + if m and target: + start = {"old": int(m.group(1)), "new": int(m.group(3))} + want = {"old": int(m.group(2) or 1), "new": int(m.group(4) or 1)} + continue + if all(len(sides[s]) >= want[s] for s in sides): + break + tag, body = line[:1], line[1:] + if tag == " ": + sides["old"].append(body) + sides["new"].append(body) + ctx += 0 if changed else 1 + elif tag in ("-", "+"): + sides["old" if tag == "-" else "new"].append(body) + changed = True + elif tag != "\\": + break + if start is None: + raise Bad("no hunk in %s" % path) + return target, {s: (start[s], sides[s]) for s in sides}, ctx + + +def landing(eas, path, side): + """(file, line) where a patch's first change sits in the commit: the hunk's `side` + lines ("old" before applying, "new" once applied) located nearest the line the + hunk header states, as git apply locates them.""" + target, sides, ctx = first_hunk(path) + stated, lines = sides[side] + have = eas.text(target).splitlines(keepends=True) + hits = [i + 1 for i in range(len(have) - len(lines) + 1) if have[i:i + len(lines)] == lines] if lines else [] + at = min(hits, key=lambda i: abs(i - stated)) if hits else stated + return target, max(at + ctx, 1) + + +def touched(paths): + """Files a set of patches change.""" + out = [] + for p in paths: + with open(p, encoding="utf-8") as f: + for line in f: + if line.startswith("+++ ") and line[4:].strip() != "/dev/null": + t = line[4:].strip() + t = t[2:] if t.startswith("b/") else t + if t not in out: + out.append(t) + return out + + +class Eas: + """Read-only views of one EAS commit, plus the working-tree apply.""" + + def __init__(self, path, rev): + self.path = os.path.abspath(path) + top = self.git("rev-parse", "--show-toplevel", check=False) + if top.returncode or os.path.realpath(top.stdout.strip()) != os.path.realpath(self.path): + raise Bad("%s is not the top of a git checkout" % path) + self.sha = self.resolve(rev) + if not self.sha: + raise Bad("%s: no commit %s" % (path, rev)) + self._files = None + self._text = {} + self._tmp = tempfile.TemporaryDirectory(prefix="eas-overlay-") + self.index = os.path.join(self._tmp.name, "index") + self.git("read-tree", "--index-output=" + self.index, self.sha) + + def git(self, *args, env=None, check=True): + e = dict(os.environ, **(env or {})) + r = subprocess.run(["git", "-C", self.path] + list(args), capture_output=True, text=True, env=e) + if check and r.returncode: + raise Bad("git %s: %s" % (" ".join(args), r.stderr.strip())) + return r + + def resolve(self, rev): + r = self.git("rev-parse", "--verify", "-q", rev + "^{commit}", check=False) + return r.stdout.strip() if r.returncode == 0 else None + + def files(self): + if self._files is None: + out = self.git("ls-tree", "-r", "-z", "--name-only", self.sha).stdout + self._files = [p for p in out.split("\0") if p] + return self._files + + def text(self, path): + if path not in self._text: + r = self.git("cat-file", "blob", "%s:%s" % (self.sha, path), check=False) + self._text[path] = r.stdout if r.returncode == 0 else "" + return self._text[path] + + def check(self, patches, reverse=False): + """git apply --check at the commit, through the scratch index.""" + args = ["apply", "--cached", "--check", "--whitespace=nowarn"] + (["-R"] if reverse else []) + patches + return self.git(*args, env={"GIT_INDEX_FILE": self.index}, check=False).returncode == 0 + + def last_commit(self, paths): + r = self.git("log", "-1", "--format=%h %s", self.sha, "--", *paths, check=False) + return r.stdout.strip() + + def worktree(self, patches, reverse=False, check=True): + args = ["apply", "--whitespace=nowarn"] + (["--check"] if check else []) + (["-R"] if reverse else []) + patches + return self.git(*args, check=False).returncode == 0 + + +def load_spec(path): + try: + with open(path, encoding="utf-8") as f: + spec = json.load(f) + except (OSError, ValueError) as e: + raise Bad("%s: %s" % (path, e)) from None + if spec.get("schema") != "k3d-gpu.eas-overlay/v1": + raise Bad("%s: schema %r, want k3d-gpu.eas-overlay/v1" % (path, spec.get("schema"))) + for row in spec["rows"]: + if row["needed_when"] not in RULES: + raise Bad("row %s: needed_when %r is not one of %s" % (row["id"], row["needed_when"], ", ".join(RULES))) + for v in row["variants"]: + v["paths"] = [patch_path(spec, f) for f in v["files"]] + for p in row["fixed_probe"] + row["moved"]: + try: + p["re"], p["glob_re"] = re.compile(p["regex"]), glob_re(p["glob"]) + except re.error as e: + raise Bad("row %s: regex %r: %s" % (row["id"], p["regex"], e)) from None + return spec + + +def patch_path(spec, name): + for d in spec["patch_dirs"]: + p = os.path.join(ROOT, d, name) + if os.path.isfile(p): + return p + raise Bad("patch %s is in none of %s" % (name, ", ".join(spec["patch_dirs"]))) + + +def rel(path): + return os.path.relpath(path, ROOT) + + +def line_of(text, pos): + return text.count("\n", 0, pos) + 1 + + +def probe(eas, row): + """(state, file, line, knob): state "fixed", "knob-missing" or None.""" + missing = None + for p in row["fixed_probe"]: + for path in eas.files(): + if not p["glob_re"].match(path): + continue + text = eas.text(path) + m = p["re"].search(text) + if not m: + continue + at = m.start("at") if "at" in p["re"].groupindex and m.start("at") >= 0 else m.start() + knob = p.get("knob") + if knob is None or re.search(r"(? %s [%s]" % (d["fact"], d["choice"], d["permalink"] if explain else d["source"]) for d in decisions] + + +def run(args, spec): + if args.policy not in POLICIES: + raise Bad("--policy %r: want one of %s" % (args.policy, ", ".join(POLICIES))) + if args.nodes < 1: + raise Bad("--nodes %d: a cluster has at least one node" % args.nodes) + facts = {"policy": args.policy, "nodes": args.nodes} + eas = Eas(args.eas, args.rev) + if args.apply and eas.sha != eas.resolve("HEAD"): + raise Bad("--apply patches the working tree, so --rev must be HEAD (%s is not)" % args.rev) + decisions = decide(spec, eas, facts) + stop = any(d["decision"] == "STOP" for d in decisions) + if args.json: + doc = {"schema": "k3d-gpu.eas-overlay-decisions/v1", "repo": spec["repo"], "rev": eas.sha, + "facts": facts, "stop": stop, "decisions": decisions} + if args.json == "-": + json.dump(doc, sys.stdout, indent=2) + print() + else: + with open(args.json, "w", encoding="utf-8") as f: + json.dump(doc, f, indent=2) + f.write("\n") + if args.json != "-": + print("# overlay at eas@%s (%s), kubernetes_cpu_policy=%s, nodes=%d" + % (eas.sha[:7], args.rev, facts["policy"], facts["nodes"])) + print("\n".join(text_lines(decisions, args.explain))) + if stop: + print("STOP: the overlay does not fit eas@%s; regenerate the rows above that say STOP. Nothing was applied." + % eas.sha[:7], file=sys.stderr) + return 3 + if args.apply: + return apply(args, eas, decisions) + return 0 + + +def apply(args, eas, decisions): + """Each chosen patch the working tree lacks, in one git apply; one it already + carries is left alone (a re-run). Any other state of a patch's files: STOP.""" + todo, done = [], [] + for d in decisions: + if d["decision"] != "apply": + continue + patches = [os.path.join(ROOT, f) for f in d["files"]] + if eas.worktree(patches): + todo += patches + elif eas.worktree(patches, reverse=True): + done += patches + else: + todo = None + break + if todo is None or (todo and not eas.worktree(todo)): + bad = touched([os.path.join(ROOT, f) for d in decisions if d["decision"] == "apply" for f in d["files"]]) + print("STOP: the working tree of %s differs from eas@%s in %s; reset it to the commit (git checkout -f %s) " + "and run again. Nothing was applied." % (args.eas, eas.sha[:7], ", ".join(bad), eas.sha[:7]), + file=sys.stderr) + return 3 + if todo and not eas.worktree(todo, check=False): + raise Bad("git apply failed in %s after its --check passed" % args.eas) + if todo: + print("applied %s to %s" % (" ".join(os.path.basename(p) for p in todo), args.eas)) + if done: + print("already applied in %s: %s" % (args.eas, " ".join(os.path.basename(p) for p in done))) + if not todo and not done: + print("nothing to apply to %s" % args.eas) + return 0 + + +def check_verified(args, spec): + bad = 0 + for v in spec["verified"]: + try: + eas = Eas(args.eas, v["rev"]) + except Bad as e: + print("FAIL %s %s: %s (fetch it)" % (v["profile"], v["rev"][:7], e)) + bad += 1 + continue + for c in v["cases"]: + decisions = decide(spec, eas, {"policy": c["policy"], "nodes": c["nodes"]}) + got = {d["id"]: d["brief"] for d in decisions} + label = "%s %s %s nodes=%d" % (v["profile"], eas.sha[:7], c["policy"], c["nodes"]) + moved = ["%s want %s, got %s" % (i, c["expect"].get(i), got.get(i)) + for i in sorted(set(got) | set(c["expect"])) if got.get(i) != c["expect"].get(i)] + summary = " ".join("%s=%s" % kv for kv in sorted(got.items())) + if moved: + bad += 1 + print("FAIL %s: %s" % (label, "; ".join(moved))) + else: + print("ok %s: %s" % (label, summary)) + for line in text_lines(decisions, args.explain): + print(" " + line) + return 1 if bad else 0 + + +def watch(args, spec): + for w in spec["watch"]: + try: + eas = Eas(args.eas, w["ref"]) + except Bad: + print("# %s: not in this clone (%s)" % (w["ref"], w["what"])) + continue + print("# %s = eas@%s (%s), kubernetes_cpu_policy=%s, nodes=%d" + % (w["ref"], eas.sha[:7], w["what"], K3D_GPU_FACTS["policy"], K3D_GPU_FACTS["nodes"])) + print("\n".join(text_lines(decide(spec, eas, K3D_GPU_FACTS), args.explain))) + return 0 + + +def main(argv): + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + ap.add_argument("--eas", required=True, help="EAS git checkout") + ap.add_argument("--rev", default="HEAD", help="commit to decide at (default HEAD)") + ap.add_argument("--policy", help="kubernetes_cpu_policy the generator chose") + ap.add_argument("--nodes", type=int, help="nodes in the cluster") + ap.add_argument("--apply", action="store_true", help="apply the chosen patches to the working tree") + ap.add_argument("--json", metavar="FILE", help="also write the decisions as JSON to FILE (- for stdout only)") + ap.add_argument("--explain", action="store_true", help="full permalinks instead of short sources") + ap.add_argument("--check-verified", action="store_true", help="check every overlay.json verified case") + ap.add_argument("--watch", action="store_true", help="decisions at every overlay.json watch ref") + ap.add_argument("--spec", default=os.path.join(HERE, "overlay.json"), help=argparse.SUPPRESS) + args = ap.parse_args(argv) + try: + spec = load_spec(args.spec) + if args.check_verified: + return check_verified(args, spec) + if args.watch: + return watch(args, spec) + if args.policy is None or args.nodes is None: + raise Bad("--policy and --nodes are required") + return run(args, spec) + except Bad as e: + print("overlay: %s" % e, file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch b/integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch new file mode 100644 index 00000000..48cf316a --- /dev/null +++ b/integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch @@ -0,0 +1,77 @@ +From 0d13b34a8ab3c5b271ab8fad3a82b726c013a4b1 Mon Sep 17 00:00:00 2001 +From: Andrew Mello +Date: Thu, 29 Oct 2026 13:35:38 -0700 +Subject: [PATCH] envoy_gateway: select control-plane nodes by label presence + (release-1.2) + +The release-1.2 form of the main-branch change in +roles/envoy_gateway/tasks/controller.yaml. On release-1.2 the EnvoyProxy +task still lives in roles/envoy_gateway/tasks/install.yaml (EAS #42 +moved it to controller.yaml on main), so the same change is cut against +that file. + +The EnvoyProxy DaemonSet pins envoy to control-plane nodes with +nodeSelector node-role.kubernetes.io/control-plane: "". That matches +kubeadm / kubespray nodes, which carry the label with an empty value, +but k3s sets it to "true" (k3s-io/k3s pkg/server/server.go, +util.ControlPlaneRoleLabelKey), so on a BYO k3s or k3d cluster the +DaemonSet schedules no pods and the gateway has no data plane. + +Use a required node affinity with operator Exists on the same key, and +set nodeSelector to null so a re-install over an older EnvoyProxy (a +JSON merge patch, which keeps every key it does not null) drops the old +selector. Behaviour on kubeadm / kubespray clusters is unchanged; the +toleration is untouched. + +Checked statically: release-1.2 pins envoy_gateway_version 1.7.1, the +same gateway-helm whose v1alpha1 EnvoyProxy CRD includes pod.affinity. +Modelling create (null pruned) and re-install over the old object +(RFC 7386 merge) both leave affinity and no nodeSelector. +--- + roles/envoy_gateway/tasks/install.yaml | 23 ++++++++++++++++------- + 1 file changed, 16 insertions(+), 7 deletions(-) + +diff --git a/roles/envoy_gateway/tasks/install.yaml b/roles/envoy_gateway/tasks/install.yaml +index 9f3d423..72c3444 100644 +--- a/roles/envoy_gateway/tasks/install.yaml ++++ b/roles/envoy_gateway/tasks/install.yaml +@@ -191,11 +191,12 @@ + # MetalLB elects at any given moment: every CP node always has a local + # envoy. Survives CP failover, no hostname pinning, no cp[0] dance. + # +-# nodeSelector keys on the standard control-plane label (empty string is +-# how kubespray / kubeadm stamp it) rather than the taint, because +-# operators can remove the taint; the label is the stable Kubernetes +-# control-plane marker. Toleration is kept for the case where the taint +-# is still in place. ++# Node affinity keys on the standard control-plane label rather than the ++# taint, because operators can remove the taint; the label is the stable ++# Kubernetes control-plane marker. It matches on the label's presence, not ++# its value: kubespray / kubeadm stamp it with "" while k3s stamps ++# it with "true", and an exact-value nodeSelector schedules no envoy there. ++# Toleration is kept for the case where the taint is still in place. + - name: "envoy_gateway | Create EnvoyProxy config (DaemonSet on control-plane nodes)" + kubernetes.core.k8s: + state: present +@@ -211,8 +212,16 @@ + kubernetes: + envoyDaemonSet: + pod: +- nodeSelector: +- node-role.kubernetes.io/control-plane: "" ++ # null drops the nodeSelector an older install left behind: ++ # this update is a JSON merge patch, which keeps omitted keys. ++ nodeSelector: null ++ affinity: ++ nodeAffinity: ++ requiredDuringSchedulingIgnoredDuringExecution: ++ nodeSelectorTerms: ++ - matchExpressions: ++ - key: node-role.kubernetes.io/control-plane ++ operator: Exists + tolerations: + - key: node-role.kubernetes.io/control-plane + operator: Exists +-- +2.53.0 + diff --git a/integrations/eat/upstream/README.md b/integrations/eat/upstream/README.md index 59f72604..e24abf01 100644 --- a/integrations/eat/upstream/README.md +++ b/integrations/eat/upstream/README.md @@ -8,7 +8,7 @@ These are drafts for Intel's repos. **None of them has been filed, opened or pus - It deletes the toolkit's own installer: `deploy-agentic-stack.sh`, `core/lib/brownfield/`, `core/playbooks/setup-bastion.yml`, `core/playbooks/deploy-istio.yml`, `core/roles/nri_cpu_balloons/`. - It moves the toolkit onto `es_auto_installer.sh install agent-toolkit` in [intel/enterprise-ai-solutions](https://github.com/intel/enterprise-ai-solutions) (EAS) and intel/enterprise-inference. - It depends on intel/enterprise-inference#22 and goes in ahead of intel/enterprise-ai-solutions#40. Both are open. - - Re-read 2026-10-29: #19 merged on 2026-10-27 into the `release-2.0.0` branch, not `main`, and no tag carries it. intel/enterprise-inference#22 merged on 2026-10-22. intel/enterprise-ai-solutions#40 was closed unmerged on 2026-10-22, and intel/enterprise-ai-solutions#66 ("Integrate Agentic Toolkit"), opened a minute before, merged on 2026-10-29. Every patch here still passes `checks/apply-check.sh` on EAS `main` `5767995`. + - Re-read 2026-10-29: #19 merged on 2026-10-27 into the `release-2.0.0` branch, not `main`, and no tag carries it. intel/enterprise-inference#22 merged on 2026-10-22. intel/enterprise-ai-solutions#40 was closed unmerged on 2026-10-22, and intel/enterprise-ai-solutions#66 ("Integrate Agentic Toolkit"), opened a minute before, merged on 2026-10-29. Every patch here still passes `checks/apply-check.sh` on EAS `main` `5767995`, and on `97f6157` (#69), the `main` head later that day. - The platform roles that matter for k3s (istio, envoy_gateway, kubernetes, nri_cpu_balloons) already live in EAS `main`. **EAS is the target for every patch here.** - There is no toolkit (EAT) patch. Its k3s platform-detection gap sits in `brownfield_deployment()`, which no entry point calls: it is sourced at [deploy-agentic-stack.sh#L631](https://github.com/intel/enterprise-agent-toolkit/blob/0ecf67a8331e8b502ef41e08bf6d2881416178c5/deploy-agentic-stack.sh#L631) and called only from inside itself, and #19 deletes the file. The issue draft lists it as an observation only. @@ -50,6 +50,18 @@ The patches carry no `Signed-off-by:`. EAS's CONTRIBUTING.md requires a DCO sign git am --signoff integrations/eat/upstream/patches/0002-eas-istio-global-platform.patch ``` +## The k3d-gpu overlay + +Until upstream carries them, k3d-gpu applies these drafts itself to the EAS checkout it installs from. [integrations/eas/overlay.json](../../eas/overlay.json) lists each patch with its variants, when it is needed (`always`, `eas_owns_nri`, `multi_node`, `never`), a probe for an upstream fix done another way, and the knob the generator writes. [integrations/eas/overlay.py](../../eas/overlay.py) decides per patch at the pinned EAS commit and prints `fact -> choice [source]`: + +- `skip`: not needed on this cluster. +- `drop-merged`: the patch reverse-applies. +- `drop-fixed`: the probe matches and the knob is still there. +- `apply`: the first variant that applies. +- `STOP`, before anything changes: it names the upstream change that moved the hunk and says to regenerate. + +The drafts here stay the only copy. The one overlay-only patch is the release-1.2 form of 0006, [integrations/eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch](../../eas/patches/0006-release-1.2-envoy-control-plane-affinity.patch); on that branch the EnvoyProxy task is still in `install.yaml`. + ## Static validation | Check | Result | @@ -63,11 +75,14 @@ git am --signoff integrations/eat/upstream/patches/0002-eas-istio-global-platfor | 0004 behaviour, [checks/eas-byo-topology.sh](checks/eas-byo-topology.sh) (stub `kubernetes.core`) | upstream: BYO 3 nodes → no labels (FAIL); patched: server `platform`, agents `inference`. BYO with one node already labelled → that node untouched; BYO with node list or patch denied → install succeeds, no labels; managed with patch denied → still fails. BYO 1 node, topology off, managed single- and multi-node all unchanged | | 0006 behaviour | the patched `EnvoyProxy` validates against the v1alpha1 CRD in gateway-helm v1.7.1, which includes `pod.affinity` and does not mark `nodeSelector` nullable | | 0006 re-install, [checks/eas-envoy-reinstall.py](checks/eas-envoy-reinstall.py) | models kubernetes.core 6.3.0 updating a custom resource (JSON merge patch, RFC 7386) over the object the unpatched task created. Without `nodeSelector: null` the old selector survives next to the affinity (FAIL); with it, gone. Fresh install: the null is pruned on create | +| 0006 release-1.2 variant, same script on EAS `release-1.2` `5c27fb3` before and after the variant | fresh install and re-install over the unpatched object both leave `nodeSelector: None` and the `Exists` affinity (2026-10-29). release-1.2 pins `envoy_gateway_version: "1.7.1"`, the gateway-helm the 0006 CRD check used | +| Overlay decisions, `checks/apply-check.sh --overlay` on a fresh clone (2026-10-29) | EAS `main` `97f6157`: 0002 and 0006 apply (0006 variant a), 0003 skips unless the policy is nri-balloons, 0004 skips on one node, 0005 always skips. `release-1.2` `5c27fb3`: the same, with 0006 variant b. Open PRs: #50 and #52 STOP 0006 (the EnvoyProxy provider moves into `_eg_k8s_provider`), #61 STOPs 0002 (`headlamp_enabled` in its `global_config.yaml` context) | How to rerun the checks (an EAS clone; `ansible-core` and `jmespath` for the topology check; PyYAML for the envoy check): ```bash integrations/eat/upstream/checks/apply-check.sh ../enterprise-ai-solutions +integrations/eat/upstream/checks/apply-check.sh --overlay # clones EAS itself; needs the network integrations/eat/upstream/checks/eas-byo-topology.sh integrations/eat/upstream/checks/eas-envoy-reinstall.py ``` diff --git a/integrations/eat/upstream/checks/apply-check.sh b/integrations/eat/upstream/checks/apply-check.sh index cc8429bc..dc9cfa57 100755 --- a/integrations/eat/upstream/checks/apply-check.sh +++ b/integrations/eat/upstream/checks/apply-check.sh @@ -4,16 +4,44 @@ # that the series applies in order with git am. The clone is copied, never # modified. # +# With --overlay it prints the decisions of integrations/eas/overlay.py, the +# overlay `k3d-gpu eas up` applies, against a real EAS clone: every +# overlay.json `verified` case at its pinned sha (fails when a recorded decision +# moved), then every `watch` ref (branch tips and open upstream PRs) for +# k3d-gpu's own facts. Without a clone it clones intel/enterprise-ai-solutions; +# a given clone is copied first. Fetches branches and PR heads, so it needs the +# network and stays out of CI. +# # Usage: integrations/eat/upstream/checks/apply-check.sh +# integrations/eat/upstream/checks/apply-check.sh --overlay [] set -uo pipefail -EAS=${1:?usage: $0 } PATCHES="$(cd "$(dirname "$0")/../patches" && pwd)" +ROOT="$(cd "$(dirname "$0")/../../../.." && pwd)" WORK=$(mktemp -d) trap 'rm -rf "${WORK}"' EXIT FAILS=0 +if [ "${1:-}" = --overlay ]; then + if [ -n "${2:-}" ]; then + cp -a "$2" "${WORK}/eas" + else + git clone -q https://github.com/intel/enterprise-ai-solutions.git "${WORK}/eas" || exit 2 + fi + git -C "${WORK}/eas" fetch -q origin '+refs/heads/*:refs/remotes/origin/*' \ + '+refs/pull/*/head:refs/remotes/pull/*/head' || exit 2 + echo "## verified pins (integrations/eas/overlay.json)" + python3 "${ROOT}/integrations/eas/overlay.py" --eas "${WORK}/eas" --check-verified || FAILS=$((FAILS + 1)) + echo + echo "## watch refs, k3d-gpu facts" + python3 "${ROOT}/integrations/eas/overlay.py" --eas "${WORK}/eas" --watch || FAILS=$((FAILS + 1)) + [ "${FAILS}" -eq 0 ] + exit +fi + +EAS=${1:?usage: $0 | --overlay []} + # fresh: a throwaway copy with the clone's HEAD checked out clean. fresh() { rm -rf "${WORK:?}/eas" diff --git a/tests/eas_test.sh b/tests/eas_test.sh new file mode 100755 index 00000000..6c27dc3c --- /dev/null +++ b/tests/eas_test.sh @@ -0,0 +1,437 @@ +#!/usr/bin/env bash +# Checks for integrations/eas without a cluster. With no argument every section +# runs; name sections to run only those (tests/eas_test.sh overlay). +# +# overlay integrations/eas/overlay.py, the k3s/k3d patch overlay, on synthetic +# git repos built from verbatim EAS files (tests/fixtures/eas/trees, +# EAS main 97f6157 and release-1.2 5c27fb3): +# - the fixtures are byte-identical to upstream (blob shas in SOURCES) +# - overlay.json resolves every patch, upstream drafts first-class, no copies +# - main: 0002 and 0006 apply, 0003 only for nri-balloons, 0004 only on +# more than one node, 0005 never; the recorded `verified` cases hold +# - patches already in the tree: drop-merged (git apply -R --check) +# - the fix done another way with the knob present: drop-fixed +# - the fix done another way with the knob renamed: STOP +# - the hunk moved as open EAS #50/#52 move it: STOP naming #50 +# - release-1.2: 0006 variant b applies, 0006a and 0005 do not +# - --apply changes the checkout only when no row STOPs, and a re-run is a no-op +# - every decision prints as `fact -> choice [source]`; --explain gives permalinks +# Needs bash, git, python3. + +set -uo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +PASS=0 +FAILS=0 +WORK=$(mktemp -d) +trap 'rm -rf "${WORK}"' EXIT + +ok() { PASS=$((PASS + 1)); echo "ok $1"; } +fail() { FAILS=$((FAILS + 1)); echo "FAIL $1"; } + +# check