diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ebc42427..81cd1259 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -269,6 +269,7 @@ jobs: - run: python3 bench/h54gpu/test_h54gpu.py - run: python3 bench/h57d/test_h57d.py - run: python3 bench/h59/test_h59.py + - run: python3 bench/h62/test_h62.py - run: python3 bench/admission/test_ctl.py - run: python3 bench/admission/test_analyze.py - run: bash bench/admission/test_run.sh diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 1d944d2f..ed652e17 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -68,14 +68,15 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H50 | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) On EAT's own target CPU (Xeon 6741P, Granite Rapids, AMX), inside EAT's banking flow, the k3d-gpu CPU stack A1 (OVMS int4 ~~with the CPU Fast Draft~~, `agent manifest`; draft OFF per [amendment A4](#crescent-amendment-a4-a1-is-k3d-gpus-best-cpu-config-for-the-xeon-2026-10-20): H34x REFUTED it on this Xeon) still delivers far more B24 tasks within the 120 s turn budget than EAT stock B1 (vLLM CPU bf16, `xeon-values.yaml` as shipped, AMX on), both arms in one hold, interleaved. H49b's comparison moved to the hardware EAT is built for. A stack-vs-stack claim (amendment 16). | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement on the Xeon. A1 within-budget success >= 0.70 (0.8). B1 >= 0.60 (0.7): with AMX bf16 its turns should fit 120 s. Paired A1 - B1 in [-0.15, +0.20] (0.6). L2 matches >= 16 pairs and >= 75% (0.6), and the gates are read first, so the expected verdict is REFUTED (0.5), INCONCLUSIVE (0.45, mostly the L2 gates), SUPPORTED (0.05). Not gated: median attempt A1/B1 <= 0.7 (0.6). | H49b's design, statistic, outcome classes, grader, gates and thresholds on `crescent`; see [H50-H52 pre-registration](#h50-h52-pre-registration-the-xeon-6741p-2026-10-20). Holds of at most 40 min until 24 pairs or 8 holds. SUPPORTED: paired difference CI lower bound >= +30 points and A1 >= 0.6, with L2, the void and idle-arm checks passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. `bench/eatflow/crescent/pairs-hold.sh H50 ` (`LIVE_MODE=pairs LIVE_CAP=2370 live.sh H50`, KV 40 GiB), `agentbench pairs-report --hyp H50 --arm-a A1 --arm-b B1`. Model, arms, sampling and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20) (Gemma 4 26B-A4B QAT; breaks direct replication of H49b). Analysis rule L3 (primary) from [amendment A3](#crescent-amendment-a3-analysis-rule-l3-for-the-crescent-pairs-rows-2026-10-20). B1 is EAT release-1.1.0's stock (vLLM CPU v0.28.0) and runs only after a stall check, per [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); H50's deploy commit, wiring smoke and pre-hold records per [amendment E3](#crescent-amendment-e3-h50s-deploy-commit-wiring-smoke-and-pre-hold-records-2026-10-29). Result ([H50 run](#h50-run-xeon-6741p-2026-10-29)): E1's stall check passed (60 of 60 requests, no stall); 24 complete pairs in 1 of the 8 permitted holds, no void pair, idle check 0.0022 CPU, balance gate passed (smallest p 0.97). Within-budget success A1 19/24 (0.79), B1 21/24 (0.875); paired difference A1 - B1 -0.083 (95% CI -0.208 to 0.000), under +30 points, so REFUTED. Not gated: median attempt A1 8.0 s, B1 12.0 s; paired J per attempt A1/B1 0.81. | REFUTED | | H50o | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) H50's comparison with arm B3, EAT's own OVMS chart (`core/helm-charts/ovms` at release-1.0.3-hotfix, stock values, `openvino/model_server` 2025.4, the closest OpenVINO/ model it pulls, `OpenVINO/Qwen3-4B-int4-ov`) instead of B1: A1 delivers far more B24 tasks within budget than B3. With H50 it splits B1 -> A1 into EAT switching its own engine (B1 -> B3) and what k3d-gpu adds on top (B3 -> A1). | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement. B3's stock graph has no tool parser (the chart's `ovms --pull` writes none) and its model reasons by default, so its tool calls arrive as text: B3 within-budget success <= 0.10 (0.7), paired A1 - B3 >= +60 points (0.65). Pair yield: a B3 attempt should end within 30-90 s (a failed tool call or a reasoned text answer), so a 40-min hold should run 15-25 pairs and 24 pairs fit 2-3 holds; L2 passes (0.6), so SUPPORTED (0.5), INCONCLUSIVE (0.45). | As H50 with arm B = B3 (same pairs, gates, thresholds, statistic, cluster per hold `k3dgpu-e-h50o`); holds of at most 40 min until 24 pairs or 6 holds. The B1 -> B3 and B3 -> A1 splits are described from the two rows' matched pairs, never gated. `bench/eatflow/crescent/pairs-hold.sh H50o `, `agentbench pairs-report --hyp H50o --arm-a A1 --arm-b B3`. Model and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20). Analysis rule L3 from [amendment A3](#crescent-amendment-a3-analysis-rule-l3-for-the-crescent-pairs-rows-2026-10-20). B3 is unchanged under [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29) (EAT release-1.1.0's OVMS chart renders the same objects); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); H50o's deploy commit, B3's records and its wiring smoke per [H50o amendment O1](#h50o-amendment-o1-deploy-commit-b3s-records-and-a-b3-wiring-smoke-2026-10-29). Result ([H50o run](#h50o-run-xeon-6741p-2026-10-29)): B3 was not ready in hold 1, so under M1's rule H50o runs no further hold and no pair ran (1 of the 6 permitted holds). On k3d, B3's PVC bound, its pod was scheduled and its init container pulled `OpenVINO/gemma-4-26b-a4b-it-int4-ov` at `4e5885c3` (every file equal to the Hub's), but OVMS 2025.4's server failed on each of its 7 starts, going into CrashLoopBackOff; the last start's recorded exit is 139 (SIGSEGV), a second into loading the model. The same server in a plain container, outside k3d, on the same revision, exited 139 the same way with and without the chart's limits. Not deliverable: EAT's OVMS chart as shipped does not serve this model. | INCONCLUSIVE | | H51 | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) Graded agents per box on the Xeon 6741P: H45's ramp, gates and scoring (B15-RO, typical load, 300 s steps, pooled p95 turn <= 10 s, success >= the arm's N=1 success - 10 points, errors <= 1%, >= 60 window requests), extended to N=16, for A1 and B1. | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement. A1 in [1, 8] (0.5). B1 in [0, 4] (0.5). B1 <= A1 (0.55). | H45's `apb` runner and scorer (`apb-score --hyp H51`, ramp 1, 2, 4, 8, 16), one fresh cluster per hold, at most 4 holds of at most 40 min. A prediction holds when the arm's bracket lies inside its range and is refuted when it lies outside; B1 <= A1 is decided only on matched load. REFUTED when any prediction is refuted, SUPPORTED when all three hold, else INCONCLUSIVE. Runs only if H50's holds end within their cap. Model and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20). B1 as H50, from [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); deploy commit, M1's ranges in the scorer, one hold per arm and the wiring smokes per [H51 amendment S1](#h51-amendment-s1-deploy-commit-scorer-ranges-ramp-per-hold-and-wiring-smokes-2026-10-29). Result ([H51 run](#h51-run-xeon-6741p-2026-10-29)): one hold per arm, 2 of the 4 permitted, none VOID. Both arms failed N=1. A1: 65 window requests, turn p95 10.41 s against the 10 s SLO, success 30/32. B1: 58 window requests (under 60), p95 11.52 s, success 27/29. Agents per box A1 0, outside its predicted [1, 4], so REFUTED. B1's 0 lies inside [0, 4], and B1 <= A1 holds on matched load (0 <= 0). | REFUTED | +| H62 | Against Intel's new default. On the Xeon 6741P, inside EAT's banking flow (B24) with Gemma 4 26B-A4B on its QAT weights, the k3d-gpu CPU stack A1 (H50's A1 unchanged: `agent manifest`, OVMS 2026.4.1 int4 on the snapped QAT IR, CPU, no draft) delivers far more tasks within the 120 s turn budget than B5, Intel's documented single-node default for its agent toolkit: EAS v1.2.0-rc.1 (`5c27fb3`) installing EAT 2.0 (`release-2.0.0` `036b662`) and enterprise-inference `release-1.1` (`170eb3a`) with Kubespray onto one node, serving its catalog entry `gemma-4-26b` (vLLM CPU v0.28.0 on KServe, NRI balloons) on the same QAT weights. Each stack runs in its own KVM guest of one shape on the same host; both are live in one hold, interleaved. Deviations from stock are named (D1-D4; D4: the harness reaches B5's workload Service, not EAS's LiteLLM gateway and guardrail). A stack-vs-stack claim (amendment 16). | Written 2026-10-29, before any H62 step and after H50 (A1 19/24, B1 21/24) and H51 were known. A1 within-budget success >= 0.70 (0.65). B5 >= 0.70 (0.6). Paired A1 - B5 in [-0.20, +0.15] (0.6). Balance gate passes (0.9). REFUTED (0.6), INCONCLUSIVE (0.35: mostly B5 not deliverable in its guest within 120 min, a failed wiring smoke or a gate), SUPPORTED (0.05). Not gated: median attempt A1/B5 <= 0.85 (0.5); A1's median attempt in its guest over H50's 8.0 s on the host in [1.0, 1.3] (0.6). | [H62 pre-registration](#h62-pre-registration-k3d-gpu-against-intels-new-default-eas-with-eat-20-on-the-xeon-6741p-2026-10-29): H50's design, statistic, gates and verdict rule (rule L3, amendment A3, seed `H62-crescent-2026-10-29`), two KVM guests on crescent (VM a 96 vCPU and 96 GiB for A1, VM b 96 vCPU and 192 GiB for B5), the harness's LiteLLM per arm on the host, holds of at most 40 min until 24 pairs or 8 holds, each hold restarting both serving pods. Setup and wiring-smoke gates (memory, disk, AMX in each guest, NFD label, B5's weights equal to the QAT checkpoint, EAS's own `validate` and gateway check) come before hold 1. SUPPORTED: paired difference CI lower bound >= +30 points and A1 >= 0.6, with every gate passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. `bench/h62/campaign.sh`; `agentbench pairs-report --hyp H62 --arm-a A1 --arm-b B5`. Re-run as a new row when an EAS release at v1.2.0 or later or an EAT release at 2.0.0 or later is tagged (`bench/h62/upstream.sh`). | PENDING | | H52 | A plain `k3d-gpu up` (no flags, the auto-adapt default) on the Xeon 6741P host as found detects and enables every engine actually present and advertises each usable one to the scheduler, with nothing missing and nothing phantom; engines present but not configured on the host are named, not advertised. | Written 2026-10-20 from an inventory read from sysfs, lspci and CPUID, not from k3d-gpu. 4 x Arc Pro B70 (xe) as `gpu.intel.com/xe: 4`; AMX TILE, BF16, INT8, FP16 and 14 AVX-512/VNNI labels; no hybrid, NUMA, NVIDIA, Gaudi or NPU objects; DSA, IAA and QAT named as present but unconfigured (0.4: `up` may not name them). | The table in [H50-H52 pre-registration](#h50-h52-pre-registration-the-xeon-6741p-2026-10-20): `k3d-gpu doctor`, `K3D_GPU_CLUSTER=k3dgpu-h52 k3d-gpu up`, `kubectl get node -o json`, `k3d-gpu test`, and one enumeration pod per B70. SUPPORTED: every row matches. REFUTED: any row does not. Run 2026-10-20 ([H52 run](#h52-run-xeon-6741p-2026-10-20)): the four B70s (`gpu.intel.com/xe: 4`, `xmx=true`), AMX (TILE, BF16, INT8, FP16) and all 14 AVX-512/VNNI labels detected and reached pods, nothing phantom; but no media engine label on `xe`, and `up` does not name the unconfigured DSA, IAA and QAT (`doctor` does). | REFUTED | | H52b | After `k3d-gpu host-setup --apply` configures the host's DSA, IAA and QAT (the fix that adds that path, merged first), a plain `k3d-gpu up` also advertises them, and each one reaches a pod; the host is then restored exactly. | Written 2026-10-20, before the fix exists. DSA, IAA and QAT allocatable > 0, each at the count host-setup reports configuring; their test pods see `/dev/dsa`, `/dev/iax`, `/dev/vfio` (0.6). The other rows read as H52's expected table, media labels included ([amendment D1](#h52-amendment-d1-the-fixes-and-h52b-2026-10-20)). | Runs after H52, once the host-setup fix is merged; same procedure plus `host-setup --apply` before `up` and its printed revert after. SUPPORTED: the three engines advertised and reached, the H52 rows unchanged, and the host inventory read back equal to the one taken before. REFUTED: any of these fails. The host-setup command (`--apply --with-kernel-wqs`), the plan read first, the counts, the plain `up`'s node image and the revert per [amendment D2](#h52-amendment-d2-h52bs-host-setup-step-on-crescent-2026-10-29); the runner and deploy commit per [amendment D3](#h52-amendment-d3-h52bs-runner-and-deploy-commit-2026-10-29). Result ([H52b run](#h52b-run-xeon-6741p-2026-10-29)): one hold at k3d-gpu 0.1.79; the plan passed D2's gate (8 commands) and applied. DSA advertised 3 of the 3 configured work queues and its pod saw `/dev/dsa`; IAA got no work queue and no resource (as D2 item 4 disclosed); QAT got 48 VFs, but its plugin crash-looped and advertised none; `gpu.intel.com/xe` read 2, not 4: the serving pairs hold about half the VRAM of all four cards, and `up`'s busy-card filter (half the VRAM) left out the two just over half (16.1, 16.2 GiB) and kept the two just under it (15.4, 15.5 GiB), which the pairs also hold, so two enumeration pods ran, not four. Media labels (`vcs` 2, `vecs` 2), AMX, the 14 AVX-512/VNNI labels, CPU 96 and no other vendor read as expected. The third inventory equals the first. Three clauses fail, so REFUTED. | REFUTED | -| H53 | Inside EAT's banking flow on the Arc Pro B70s with Qwen3.8-27B (thinking off), k3d-gpu's own GPU choice AG (OVMS int4 on one card, `agent manifest`, auto device) delivers far more B24 tasks within the 120 s turn budget than B2, "EAT PR #21 nearest profile (Qwen3.6-27B) on Qwen3.8 weights, fp16" (vLLM XPU v0.28.0, TP 4). A stack-vs-stack claim. | Written 2026-10-20, before any H53 measurement. AG >= 0.70 (0.6); B2 boots by ladder rung (b) (0.6); B2 >= 0.50 (0.5); block mean AG - B2 in [-0.15, +0.30] (0.5). INCONCLUSIVE (0.6), REFUTED (0.3), SUPPORTED (0.1). | Solo holds in the order AG, B2, B2, AG, AG, B2 (up to 8), blocks = hold pairs (2k-1, 2k), block-level inference: mean of per-block d over >= 3 counted blocks, t-interval. SUPPORTED: lower bound >= +30 points and AG >= 0.6, L3 gates passed. REFUTED: upper bound < +30 points. Else INCONCLUSIVE. `bench/eatflow/crescent/solo-hold.sh N ARM`, `agentbench solo-report --hyp H53 --arm-a AG --arm-b B2`. See [H53 pre-registration](#h53-pre-registration-eats-banking-flow-on-arc-pro-b70-2026-10-20). | PENDING | +| H53 | Inside EAT's banking flow on the Arc Pro B70s with Qwen3.8-27B (thinking off), k3d-gpu's own GPU choice AG (OVMS int4 on one card, `agent manifest`, auto device) delivers far more B24 tasks within the 120 s turn budget than B2, "EAT PR #21 nearest profile (Qwen3.6-27B) on Qwen3.8 weights, fp16" (vLLM XPU v0.28.0, TP 4). A stack-vs-stack claim. | Written 2026-10-20, before any H53 measurement. AG >= 0.70 (0.6); B2 boots by ladder rung (b) (0.6); B2 >= 0.50 (0.5); block mean AG - B2 in [-0.15, +0.30] (0.5). INCONCLUSIVE (0.6), REFUTED (0.3), SUPPORTED (0.1). | Solo holds in the order AG, B2, B2, AG, AG, B2 (up to 8), blocks = hold pairs (2k-1, 2k), block-level inference: mean of per-block d over >= 3 counted blocks, t-interval. SUPPORTED: lower bound >= +30 points and AG >= 0.6, L3 gates passed. REFUTED: upper bound < +30 points. Else INCONCLUSIVE. `bench/eatflow/crescent/solo-hold.sh N ARM`, `agentbench solo-report --hyp H53 --arm-a AG --arm-b B2`. See [H53 pre-registration](#h53-pre-registration-eats-banking-flow-on-arc-pro-b70-2026-10-20). Node image, deploy commit, order and the restore check per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); the solo runner fix, setup, the wiring smokes, the window and its futility stop per [H53 amendment R1](#h53-amendment-r1-the-solo-runner-fix-setup-wiring-smokes-and-the-gpu-window-2026-10-30); tools per [amendment E4](#crescent-amendment-e4-kubectl-v1372-helm-v431-and-uv-0136-in-the-crescent-workspace-2026-10-29). Result ([H53 run](#h53-run-arc-pro-b70-2026-10-30)): no hold ran, 0 of the 8 permitted. AG's wiring smoke passed (OVMS on one card, warm-up answered). B2 needed rung (b) (`setvars.sh` is not in the v0.28.0 image), then failed in its first cross-card collective: oneCCL opens `/dev/dri/by-path`, which the k3d node and pod lack. No further rung applies, so B2 is not deliverable and H53 is INCONCLUSIVE by its pre-registered rule. | INCONCLUSIVE | | H54a | On the Xeon 6741P, Gemma 4 26B-A4B (QAT weights) single-stream: the best optimized CPU arm, chosen by the pre-registered source and speed rule, decodes at N=1 at least 3x faster than EAT stock B1g (vLLM CPU v0.24.0 bf16, `xeon-values.yaml`'s Gemma entry on the QAT bf16 weights), with a 50-item quality spot-check no more than 10 points below B1g's. A microbench, not an agent claim. | Written 2026-10-20, before any H54 measurement. N=1 median decode tok/s: B1g in [4, 15] (0.6); Lg (llama.cpp CPU, Google QAT Q4_0) in [25, 70] (0.6); A1g (k3d-gpu auto, OVMS CPU, QAT-matched IR) in [15, 60] (0.5), A1g/A1o in [0.85, 1.02] (0.6); Vq (vLLM CPU v0.32.0 bf16) in [6, 18] (0.5), Vq-mtp/Vq in [1.4, 2.5] (0.5); Lg-mtp/Lg in [1.2, 2.2] (0.4); Sg (SGLang CPU AMX) deliverable (0.6). The chosen arm is Lg or Lg-mtp (0.55). B3g not deliverable (0.8). SUPPORTED (0.65), INCONCLUSIVE (0.25), REFUTED (0.1). Not gated: TTFT on a 1,000-token prompt, aggregate tok/s at N=4 and 16 (A1g N=16/N=1 <= 1.5, 0.5), J/token. | [H54 pre-registration](#h54-pre-registration-gemma-4-26b-a4b-qat-and-qwen38-microbenches-on-crescent-2026-10-20): `bench/llm_load.py` per arm (N=1: 8 requests, N=4: 16, N=16: 32, 256 tokens, greedy, `ignore_eos`), `bench/energy.sh` RAPL per phase, GSM8K items 0-49 graded by exact match, one arm serving at a time, at most 3 holds of at most 40 min. Statistic: ratio of the chosen arm's median N=1 decode to B1g's, 95% bootstrap CI over requests. SUPPORTED: CI lower bound >= 3.0 and the quality gate passed. REFUTED: CI upper bound < 3.0, or every arm with a point ratio >= 3.0 fails the quality gate. Otherwise INCONCLUSIVE (also: B1g not deliverable, or a void phase not re-run). Run 2026-10-21, 3 holds ([H54a run](#h54a-run-xeon-6741p-2026-10-21)): stock B1g stalled in both its holds (vLLM v0.24.0 `EngineDeadError` mid-decode, cause open; no scored phase, 0/50 GSM8K, all errored), so `h54.py score-a` has no base and no chosen arm. Measured: Lg 48.74, Lg-mtp 54.52, A1g 23.25, A1o 21.73 tok/s N=1 median, GSM8K 50, 50, 49, 49 of 50; each scored phase exists only under amendment R4 (its own foreign CPU 4.4-4.8 CPU-s/s). Vq, Vq-mtp, Vi, Sg, B3g not run (cap). Descriptive, not scored: Lg-mtp over B1g's 2 hold-2 rates 3.02 ~~(CI 2.90-3.06)~~ (descriptive only: n=2 base rates from an engine that died minutes later, no CI; correction 2026-10-28, [H54a run](#h54a-run-xeon-6741p-2026-10-21)). | INCONCLUSIVE | | H54q | QAT matters at int4 on this model: on GSM8K items 0-199, the QAT-matched OpenVINO IR OVQ (INT4_SYM, group 32, built from Google's QAT bf16 weights) is non-inferior to the bf16 QAT reference R within 5 points, and the published base-model IR OVB (`OpenVINO/gemma-4-26b-a4b-it-int4-ov`; ~~INT4_ASYM group 128~~ AWQ INT4 asymmetric group 64, router layers unquantized, per its `openvino_config.json`: [note N1](#h54-note-n1-what-ovb-is-2026-10-21)) scores below OVQ. | Written 2026-10-20, before any measurement. R accuracy in [0.85, 0.96] (0.6). OVQ - R in [-0.03, +0.02] (0.6). GGUF Q4_0 (Lg) - R in [-0.03, +0.02] (0.6). OVB - R in [-0.07, 0] (0.5). The OVB < OVQ part resolves on 200 items (0.2). INCONCLUSIVE (0.65), SUPPORTED (0.2), REFUTED (0.15). | As H54a's quality protocol on items 0-199 of `openai/gsm8k` `main` test @ `740312ad`; items 0-49 come from H54a's runs of the same arm configs; ~~at most 1 hold of at most 40 min for the rest~~ the rest in at most 3 holds of at most 40 min, R answering all 200 items because H54a never ran its config ([amendment Q1](#h54-amendment-q1-h54qs-reference-arm-hold-cap-and-order-2026-10-29); runner: [amendment Q2](#h54-amendment-q2-h54qs-runner-2026-10-29)). Paired bootstrap 95% CI over items (10,000 resamples, seed 1). SUPPORTED: CI lower bound of OVQ - R >= -0.05 and CI upper bound of OVB - OVQ < 0. REFUTED: CI upper bound of OVQ - R < -0.05, or CI lower bound of OVB - OVQ > 0. Otherwise INCONCLUSIVE. Result ([H54q run](#h54q-run-xeon-6741p-2026-10-29)): 3 holds of the 3 permitted. R answered items 0-199, and OVQ, OVB and Lg answered items 50-199, in holds 1 and 2, with no errored item. Items 0-49 of OVQ, OVB and Lg are H54a's (Q1 item 2). Hold 3 ran only Vi, which was not deliverable (vLLM v0.32.0 could not load the AutoRound checkpoint). Correct of 200: R 194 (0.970), OVQ 191, OVB 192, Lg 193. OVQ - R -0.015 (95% CI -0.035 to 0.000): its lower bound clears -0.05. OVB - OVQ +0.005 (CI -0.015 to +0.025): its upper bound is not below 0, and its lower bound is not above 0. Neither the SUPPORTED bounds nor a REFUTED bound hold, so INCONCLUSIVE, the expected verdict (0.65). | INCONCLUSIVE | | H54b | On Arc Pro B70 with Qwen3.8-27B single-stream: the best optimized arm on ONE card (k3d-gpu auto OVMS int4, Intel's llm-scaler vLLM, or llama.cpp SYCL, with and without each one's documented MTP) decodes at N=1 at least 2x faster than EAT #21's XPU path B2q on TWO cards (vLLM XPU v0.28.0, the `xpu-qwen36-27b` profile's flags on the Qwen3.8-27B weights, TP 2). | Written 2026-10-20, before any measurement. N=1 median decode tok/s: X1q (OVMS GPU int4) in [15, 28] (0.5), X1q-mtp/X1q in [1.3, 2.2] (0.4); Sq (llm-scaler 1.5, online FP8, TP 1) deliverable (0.5), in [10, 18]; Sq-mtp/Sq in [1.4, 2.3] (0.5); Lsq (llama.cpp SYCL, Intel Q4_K_M) in [12, 24] (0.5); B2q deliverable at TP 2 (0.4), in [5, 12] (0.5). Chosen/B2q >= 2 (0.6). Cards free for a window (0.5). INCONCLUSIVE (0.6), SUPPORTED (0.3), REFUTED (0.1). | H54a's protocol on the B70s, cards only when `~/k3dgpu-bench/gpu-state.txt` lists them free; at most 2 holds of at most 40 min. Same statistic and quality gate (against B2q, or the official-weights arm Sq when B2q is not deliverable, for the gate only). SUPPORTED: CI lower bound of chosen one-card arm / B2q >= 2.0 and the gate passed. REFUTED: CI upper bound < 2.0. Otherwise INCONCLUSIVE (also: B2q not deliverable, or no free card before the campaign ends). Gemma 4 QAT on one B70 (X1g, X1o, B2g, Sgg) is described if hold time remains, never gated. Run 2026-10-21, 2 holds ([H54b and H54c run](#h54b-and-h54c-run-arc-pro-b70-2026-10-21)): INCONCLUSIVE, not delivered as pre-registered: the pinned "one card" pods saw all 4 cards. B2q, Sq, Sq-mtp (Qwen bf16, 80.6 GB with its guard) and Rq were not deliverable (disk); X1q and X1q-mtp failed on two `h54b.sh` runner bugs before any engine started; Lsq ran, but its pod listed every render node and llama.cpp split the model over all four B70s, two of them serving the live `qwen38-a-hr`, so no one-card arm exists. Lsq-mtp was not run: its GGUF carries the MTP (`nextn`) tensors, but G1's runner has no such arm; nor were the described-only Gemma arms. `score.py`: "no gate reference (B2q and Sq missing)". Descriptive only, a 4-card run and not a one-card or B70 result: Lsq 27.3 tok/s N=1 median, GSM8K 48/50. | INCONCLUSIVE | | H54c | Hybrid placement ("use the fast DRAM"): llama.cpp with Gemma 4 26B-A4B QAT Q4_0, routed experts on the Xeon and everything else on one B70 (Hg, `-ngl 99 -ot exps=CPU`), decodes at N=1 within 25% of a pre-stated byte-split model built from the same image's full-CPU (Cg) and full-GPU (Gg) runs. Qwen3.8-Flash-Next falls back to Gemma 4 QAT (reasons in the pre-registration). | Written 2026-10-20, before any measurement. Model: t_Hg = t_Cg x (B_exp / B_tot) + t_Gg x (B_rest / B_tot) + 60 x 30 us, bytes from the GGUF tensor table. Within 25% (0.45). Hg < Gg (0.8). Hg / Cg in [0.8, 1.6] (0.5). Gg in [30, 90] tok/s (0.4). SUPPORTED (0.45), INCONCLUSIVE (0.3), REFUTED (0.25). | One hold of at most 40 min on one free card: Cg, Gg, Hg in that order, then Gg again (drift check), all in `ghcr.io/ggml-org/llama.cpp:server-intel-b11749`, H54a's N=1 phase only. SUPPORTED: abs(Hg measured / predicted - 1) <= 0.25. REFUTED: > 0.40. Otherwise INCONCLUSIVE (also: any of the three arms not delivered, or the two Gg runs differ by > 10%). Run 2026-10-21, 1 hold ([H54b and H54c run](#h54b-and-h54c-run-arc-pro-b70-2026-10-21)): INCONCLUSIVE, not delivered as pre-registered: the pinned "one card" pods saw all 4 cards. Gg, Hg and Gg2 split their GPU layers over four B70s (two serving the live `qwen38-a-hr`), and Cg (`-ngl 0`) also allocated about 0.5 GB on one of those cards, so t_Cg and the prediction are confounded. The verdict does not come from `predict.py`'s band test, whose inputs are these 4-card and confounded runs. Descriptive only, not one-card or hybrid results: Cg 36.8, Gg 58.0, Hg 56.3, Gg2 57.4 tok/s N=1 median. | INCONCLUSIVE | -| H54d | Inside EAT's banking flow with Gemma 4 26B-A4B (QAT), the best H54a CPU arm A* delivers far more B24 tasks within budget than EAT stock B1g, under H50's design, statistic, gates and thresholds (rule L2, amendment C1). Runs only when A* is not H50's A1 config. | Written 2026-10-20, before any measurement. A* within-budget success >= 0.60 (0.55). B1g >= 0.50 (0.6). Paired A* - B1g in [-0.10, +0.30] (0.6). A* serves B24's tool calls without a parser change (0.6). Fewer than 16 matched pairs in 3 holds (0.5). REFUTED (0.4), INCONCLUSIVE (0.5), SUPPORTED (0.1). | H50's procedure with arm A = A*, arm B = B1g; at most 3 holds of at most 40 min, until 24 pairs. SUPPORTED: paired difference CI lower bound >= +30 points and A* >= 0.6, L2, void and idle-arm checks passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. Not run (and so recorded) when A* is H50's A1 config: H50 answers it. Not run 2026-10-21, no hold used ([H54d: not run](#h54d-not-run-2026-10-21)): A* is undefined (H54a chose no arm), B1g stalled in both H54a holds, and H54q has not run; the caps rule makes a row with its stock or chosen arm missing INCONCLUSIVE. | INCONCLUSIVE | +| H54d | Inside EAT's banking flow with Gemma 4 26B-A4B (QAT), the best H54a CPU arm A* delivers far more B24 tasks within budget than EAT stock B1g, under H50's design, statistic, gates and thresholds (rule L2, amendment C1). Runs only when A* is not H50's A1 config. | Written 2026-10-20, before any measurement. A* within-budget success >= 0.60 (0.55). B1g >= 0.50 (0.6). Paired A* - B1g in [-0.10, +0.30] (0.6). A* serves B24's tool calls without a parser change (0.6). Fewer than 16 matched pairs in 3 holds (0.5). REFUTED (0.4), INCONCLUSIVE (0.5), SUPPORTED (0.1). | H50's procedure with arm A = A*, arm B = B1g; at most 3 holds of at most 40 min, until 24 pairs. SUPPORTED: paired difference CI lower bound >= +30 points and A* >= 0.6, L2, void and idle-arm checks passed. REFUTED: CI upper bound < +30 points. Otherwise INCONCLUSIVE. Not run (and so recorded) when A* is H50's A1 config: H50 answers it. Not run 2026-10-21, no hold used ([H54d: not run](#h54d-not-run-2026-10-21)): A* is undefined (H54a chose no arm), B1g stalled in both H54a holds, and H54q has not run; the caps rule makes a row with its stock or chosen arm missing INCONCLUSIVE. Re-checked 2026-10-29 ([H54d: re-checked, still not run](#h54d-re-checked-still-not-run-2026-10-29)): still not run, no hold used. H54q has since run. A* is still undefined, and B1g stays at v0.24.0 ([amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29)), whose stall is vllm-project/vllm#45844 ([note N2](#h54-note-n2-why-b1g-stalled-unscored-diagnostic-2026-10-29)). An amendment that defines A* and moves B would now follow H50's measured B1 outcome, so none is written. 1 of H54d's 3 holds remains after [amendment Q1](#h54-amendment-q1-h54qs-reference-arm-hold-cap-and-order-2026-10-29). | INCONCLUSIVE | | H58 | The zero-touch thesis on the Xeon 6741P with four Arc Pro B70s. From a clean state, k3d-gpu brings up EAT's banking flow on Gemma 4 26B-A4B-it with no flag, no manual step and no edit. It detects and advertises every engine H52 and H52b found. It places each model by its own memory fit. Its result is within tolerance of the best hand-tuned arm on the same host. Under a controlled host memory squeeze, its guard backs off and nothing outside its own pods is OOM-killed. | Written 2026-10-20, before any H58 run and before the model optimizer (PR #163) and the fit-and-guard layer (branch `claude/fit-and-guard`) are on main. (a) Flags beyond the documented commands: 0 (0.8). (b) Manual steps or edits: 0 (0.6). (c) Decode tok/s >= 0.90x the best hand-tuned arm on this host (0.6), and within-budget B24 success within 5 points of it (0.7). (d) Every engine H52/H52b found is detected and advertised (0.75). (e) Qwen3.8-Flash-Next W4A16 (Intel AutoRound) lands on an experts-in-DRAM placement, and Gemma 4 and Qwen3.8-27B land on the GPU or the CPU, each as its fit decides, with no flag (0.5). (f) Under the squeeze: 0 OOM kills outside our pods (0.85), and the guard lowers admission (0.6). All six together: SUPPORTED (0.25), REFUTED (0.6), INCONCLUSIVE (0.15). | See [H58 pre-registration](#h58-pre-registration-the-zero-touch-thesis-on-crescent-2026-10-20) and [amendment P1](#h58-amendment-p1-the-model-pins-2026-10-29) (the model pins). Runs last on crescent, after #163 and the fit-and-guard layer merge, with their commits recorded. SUPPORTED: (a) through (f) all hold. REFUTED: any clause fails on a valid run. A manual step found is a REFUTED run. It is fixed test-first, and the re-run is a new run with its own section. INCONCLUSIVE: no valid baseline for (c), or the run is not deliverable for reasons outside k3d-gpu. | PENDING | | H60 | On the Xeon 6741P, Qwen3.8-Flash-Next (125B MoE, 6B active, a 51B-parameter n-gram table) runs entirely from host DRAM through llama.cpp, and reads the same share of its DRAM bandwidth ceiling as the Gemma 4 QAT GGUF on the same llama.cpp build: with e = N=1 median decode / (STREAM triad 305 GB/s / bytes read per token from the GGUF's tensor table), Flash-Next's e over a same-hold Gemma control's e (H54's Lg) lies within [0.7, 1.3] (without a valid control, e within 0.38 x [0.7, 1.3], H54a Lg's). A microbench of DRAM-resident serving, not an agent claim. | Written 2026-10-21, before any H60 measurement. Bytes per token 5.94 GB (all tensors but the lookup tables, 10/512 of the routed experts), ceiling 51.3 tok/s at 305 GB/s (53.9 at 320). e_fn 0.30 (0.18-0.45), N=1 decode 15 tok/s (9-23), control e_lg 0.38, ratio 0.79 (0.5-1.15): the Gated DeltaNet and hyper-connection compute per token is heavier than Gemma's. GSM8K 0-49 >= 40/50 (0.7). SUPPORTED 0.45, REFUTED 0.35, INCONCLUSIVE 0.2. | [H60 pre-registration](#h60-pre-registration-qwen38-flash-next-from-host-dram-on-the-xeon-6741p-2026-10-21): `bench/flashnext/h60.sh` inside `bench/xhost/hold.sh`, H54's one-arm protocol (idle, 3 warm-ups, N=1 8 x 256, N=4 16 x 256, TTFT 4 x 16 on H54's 1,000-token prompt, GSM8K 0-49, RAPL per phase, H54's void rule and one re-run), after a same-hold control (Gemma 4 QAT Q4_0 GGUF, idle, warm-ups, N=1 8 x 256), scored by `bench/flashnext/h60.py score`: 95% bootstrap CI of the ratio inside the band SUPPORTED, wholly outside REFUTED, else INCONCLUSIVE (also when N=1 is void, the PSI guard stops the hold, or GSM8K is below 25/50). Both arms run llama.cpp v0.8.0 (`server-b11960`), per [amendment V1](#h60-amendment-v1-llamacpp-v080-for-both-arms-2026-10-29). | PENDING | | H57da | On an AMX-less client CPU (i9-13900T, the 16 P-core CPUs, AVX2 and VNNI, no AVX-512), with the Gemma 4 26B-A4B QAT family, k3d-gpu's own CPU pick Od (`agent manifest` auto on the P-cores: OVMS 2026.4.1, `OpenVINO/gemma-4-26b-a4b-it-int4-ov`) is the fastest single-stream arm: no other arm passing the quality gate (llama.cpp with Google's QAT Q4_0 GGUF, with or without the QAT MTP drafter) decodes at N=1 5% or more faster. A microbench, not an agent claim. | Written 2026-10-20, before any H57d measurement. N=1 median decode tok/s: Od in [8, 20] (0.5); Ld (llama.cpp Q4_0, 8 threads) in [12, 30] (0.5); Ld/Od in [1.1, 1.8] (0.5); every arm's GSM8K 0-49 score >= 0.80 (0.6). REFUTED (0.55), SUPPORTED (0.25), INCONCLUSIVE (0.2). Not gated: TTFT on a 1,000-token prompt, aggregate tok/s at N=4, package J/token gross and net of idle. | [H57d pre-registration](#h57d-pre-registration-gemma-4-26b-a4b-qat-on-the-amx-less-client-cpu-i9-13900t-2026-10-20): H54's measurement protocol per arm (`bench/llm_load.py`, 256 tokens, greedy, `ignore_eos`; RAPL per phase; GSM8K items 0-49 graded as H54), one arm serving at a time, arm order rotated over 3 performance holds, quality in a 4th, at most 5 holds of at most 45 min. Statistic: R = median N=1 decode of the fastest other gate-passing arm X / Od's, over the pooled requests of the 3 holds, 95% bootstrap CI (H54's). REFUTED: Od fails the quality gate, or CI lower bound >= 1.05 and X faster than Od in each of the 3 holds. SUPPORTED: Od passes the gate and CI upper bound < 1.05. INCONCLUSIVE otherwise, or when Od or every other arm is not delivered, or the load gate fails. Run 2026-10-21 ([H57d run](#h57d-run-i9-13900t-2026-10-21)): Ld-mtp/Od 0.785 (95% CI 0.692-1.561), per hold 0.72, 0.79, 1.44; Od 6.90, Ld 3.50, Ld-mtp 5.41 tok/s pooled N=1; all 50/50 on GSM8K; load matched (1.80 <= 2.0). The strict-pinned llama.cpp arms tracked host load 2.3-2.7x, Od 1.3x. | INCONCLUSIVE | @@ -4202,6 +4203,361 @@ are byte-identical across the two versions. **Disclosed.** Written with the predictions of the covered rows in view and before any of their data existed. +### H62 pre-registration: k3d-gpu against Intel's new default, EAS with EAT 2.0, on the Xeon 6741P (2026-10-29) + +Written 2026-10-29 and committed with its runner (`bench/h62/`) before any H62 step. No guest +exists on crescent, nothing has been installed there for H62, and nothing below is a measurement. +Crescent was only read, over read-only ssh, at 2026-10-30T00:04Z. The predictions in the row are +never edited. A protocol change is a dated amendment, merged before the data it governs. + +**Why.** + +- The box owner, 2026-10-29: EAS becomes Intel's default once full support lands, so k3d-gpu + should be benchmarked against it too. +- H50, H50o and H51 compared k3d-gpu's CPU stack with EAT release-1.1.0's stock arms, deployed by + EAT's own release-1.x charts. H50 is REFUTED (A1 19/24 within budget, B1 21/24) and so is H51. +- EAT 2.0 has no installer of its own. EAT `release-2.0.0`'s README ("Installing this release") + says the Agent Toolkit installs through the Intel AI for Enterprise Solutions (EAS) installer: + `./es_auto_installer.sh configure`, `init agent-toolkit`, `install agent-toolkit`. +- Intel's three sources name three EAS clones. That README clones EAS with no `-b`, which is + main (`97f6157`), whose layer manifest names EI `main` and EAT branch `agentic-toolkit-migration` + (`fb59459`), not the release. `docs/single-node-deployment.md` clones `-b release-1.2`. The rc.1 + release notes clone `-b v1.2.0-rc.1` and say it replaces the guide's clone. B5 follows the + release notes; `release-1.2`'s head is the same commit (read 2026-10-29). +- H62 asks H50's question against that path: Intel's documented single-node default, run as + documented, at pinned commits. + +**Heads, read 2026-10-29 (`gh api`, `git ls-remote`, the Hub API, the registries).** + +- EAS: main `97f6157d6a72f7173c698c474d4e51d2c79f1fd6`. `v1.2.0-rc.1` is a pre-release + (2026-10-27) at `5c27fb39a95eecec41aa4e29a1188984e9c726f3`, which is also the head of branch + `release-1.2`. Its `VERSION` file reads 1.1.0, a known issue its release notes name. No EAS + v1.2.0 tag exists. +- EAT: `release-2.0.0` (EAT #19 merged into it) is at `036b66234801e9bd950afd90830821859780d668`, + with no tag. EAT's newest tag is `release-1.1.0` (`0c726f57`). +- enterprise-inference (EI): `release-1.1` is at `170eb3abb1956de0787727f21e356aef2b13ff24`. The + rc.1 release notes' component table names exactly these three commits. +- Hugging Face: `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized` head is still + `f1e06dc520982d9b9edd76859fdb7ab209449949` (amendment M1's revision). `google/gemma-4-26B-A4B-it` + head is `4d7ae4984b7db7de8f8457170b3f1a419ee76d52`. Neither is gated. +- `vllm/vllm-openai-cpu:v0.28.0` serves index digest + `sha256:197354e6475e638eb5ccca09beb078488046a82de0e204e7429bf1a804df9647`, the image of + amendment E1's B1. + +**Intel's documented default for one Xeon box, read from source.** + +- The install path. EAT `release-2.0.0` `docs/single-node-deployment.md`, Step 1: + `git clone -b release-1.2` of EAS, `./es_auto_installer.sh configure`, `init agent-toolkit`, the + edits to `env/local/global_config.yaml` and `env/local/config.agent-toolkit.yaml`, then + `install agent-toolkit` ("Estimated time: 20-40 minutes on a fresh node", line 112, and "the vLLM + pod may take an additional 10-15 min to pull the model weights", line 127). The rc.1 release notes + say to `git clone -b v1.2.0-rc.1` and follow that guide from Step 1. +- Kubernetes is provisioned by the installer by default (lines 36-37). The README's "Deployment + Platform" section names "Kubernetes (Default)". EAS's shipped inventory is "Option A: Localhost + (default, single-node PoC)" with `ansible_connection: local` (`inventory/hosts.yaml` lines 8-22). + EAS v1.2.0-rc.1 installs Kubernetes 1.34.3 with Kubespray v2.30.0 and containerd + (`roles/kubernetes/defaults/main.yaml` lines 18, 25 and 94). +- EAT's Docker Compose path (`docker/deploy.sh`) is for "Development, demos, single machine" in + the README's table. It is not EAS and not Intel's default, so it is not an arm. +- The serving default (EI `release-1.1`, `model_manager/models.yaml`): + - `defaults.require_amx: true` (line 20), rendered as a required node affinity on + `feature.node.kubernetes.io/cpu-cpuid.AMXBF16=true` (`core/renderer.py` lines 90-95). + - A KServe `LLMInferenceService` (KServe v0.18.0, EI's `roles/kserve`), behind Envoy's AI + gateway with endpoint picking, registered in LiteLLM. + - Entry `gemma-4-26b` (lines 360-380): `google/gemma-4-26B-A4B-it`, `cpu: auto`, memory 128Gi, + vLLM CPU `server_version` 0.28.0, args `--block-size 128`, bf16, `mp`, chunked prefill, + `--enforce-eager`, prefix caching, tool parser gemma4, TP 1 and no `--max-model-len`; env + `VLLM_CPU_SGL_KERNEL` "0" and `VLLM_CPU_KVCACHE_SPACE` "40". + - The runtime's common env adds `VLLM_CPU_NUM_OF_RESERVED_CPU` "1" and `KMP_BLOCKTIME` "1" + (lines 557 and 560); the `llm` category adds `--max-num-seqs 128 --max-num-batched-tokens + 2048` (line 585). The image is `docker.io/vllm/vllm-openai-cpu:v0.28.0`, by tag (line 571). + - Under NRI balloons (EAS's default `kubernetes_cpu_policy`), `cpu: auto` takes the free + physical cores of one NUMA node (single-node-deployment.md, Step 1b). +- Gemma 4 26B-A4B is in the stock catalog (`gemma-4-26b`, row 26 of the guide's model table) but + is not the default model: EAT's `config.yaml` line 35 autodeploys `qwen2-5-coder-14b`. With the + guardrail on (the default with `auth_provider: litellm`, `config.yaml` line 26), Qwen3Guard-Gen-0.6B + deploys too (8 CPU, 16Gi, `models.yaml` lines 412-430). + +**Why both arms run in KVM guests.** + +- Kubespray on crescent's host would remove Docker. With `container_manager: containerd` and + docker.service running, Kubespray v2.30.0 runs its "Uninstall docker" block + (`roles/container-engine/validate-container-engine/tasks/main.yml`): Docker's reset kills every + container, runs `docker ps -aq | xargs -r docker rm -fv`, stops docker and containerd, removes + the packages and deletes `/var/lib/docker`. That would remove the two serving pairs, which the + box owner's grant says never to remove. +- EAS's `configure` refuses Ubuntu other than 22.04 and 24.04 (`es_auto_installer.sh` lines + 369-378). Crescent runs Ubuntu 26.04.1. +- A bring-your-own kubeconfig onto a k3d cluster is not stock either: unpatched EAS does not + deploy on k3s or k3d (k3d-gpu's overlay patches 0002 and 0006, PRs #248 and #260). That path + is k3d-gpu's own `eas up`, a different arm (A-EAS, below). +- So B5 runs Intel's default path unmodified, on a supported OS, in a disposable KVM guest on + crescent (Ubuntu 24.04), isolated from the host's Docker serving. +- A1 runs in a guest of the same shape. A1 on bare metal against B5 in a guest would hand + k3d-gpu an edge that comes from virtualization, not from either stack. One shape gives + neither arm bare metal; it does not show the guest costs both engines alike. vLLM CPU binds + one OpenMP thread per core and OVMS schedules its own pool, and both run on unpinned vCPUs. + So the VM tax is described on both sides, never gated: A1 in its guest against H50's A1 on + the host, and B5 in its guest against H50's B1 on the host (the same vLLM CPU v0.28.0 image + and QAT weights under another deployment, so that side reads the guest and the deployment + together). + +**Arms.** + +- **A1**: H50's A1 unchanged (amendments M1 and A4), in VM a. + - `k3d-gpu agent manifest` with `K3D_GPU_AGENT_DEVICE=CPU`, OVMS 2026.4.1, the snapped QAT IR + (`k3dgpu_gemma-4-26B-A4B-it-qat-int4-sym-g32-ov`, INT4 symmetric, group 32), tool parser + gemma4, no reasoning parser, no draft, no autoquant, as `live.sh`'s `gemma4` model set gives + it. + - The IR reaches the guest read-only (virtfs) and is checked against its `manifest.json` before + the copy into the node, as H50's was. + - `k3d-gpu up` runs in the guest at the deploy commit, with H50's node image + (`cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa...`) and amendment C1's kubelet eviction + thresholds. Docker CE 5:29.9.2 (containerd.io 2.4.2) comes from Docker's apt repository, + pinned, its key's fingerprint (`9DC85822...0EBFCD88`) checked first. k3d v5.9.0 and kubectl + v1.37.2 are the crescent workspace's checked binaries (amendment E4). + - VM a has no GPU, so k3d-gpu's auto device would pick the CPU too; the device stays named, as + in H50. Amendment A4's record runs once at setup: `k3d-gpu plan --device cpu --model + google/gemma-4-26B-A4B-it --explain` in VM a. A configuration other than A1's is recorded and + does not change A1. +- **B5**: Intel's documented single-node install in VM b, with deviations D1-D4 only. The steps, + in order: + 1. The node kernel settings of EAT's `docs/prerequisites.md` ("Node Kernel Settings"), verbatim: + `fs.inotify.max_user_instances = 8192` and `fs.inotify.max_user_watches = 1048576`. + 2. `git clone -b v1.2.0-rc.1`; the head must be `5c27fb3`. + 3. D1. + 4. `./es_auto_installer.sh configure --force`. `--force` is the installer's documented answer to + configure's one y/N prompt ("required in CI, where there is no terminal"); configure's own + steps are unchanged. + 5. `init agent-toolkit`. The two cloned layers must be at `170eb3a` and `036b662`. + 6. The one documented edit: `llm_services_autodeploy_models: ["gemma-4-26b"]` in + `env/local/config.agent-toolkit.yaml` (a value of the guide's model table). Then D2. + 7. `install agent-toolkit`, with `HF_TOKEN` unset (neither model is gated; recorded as set or + unset). + 8. Wait for the gemma-4-26b workload pod to be Ready. + + Left at EAS's and EAT's defaults: Kubespray onto localhost, NRI balloons, Istio ambient, + MetalLB, Envoy's AI gateway, observability, LiteLLM with the content-safety guardrail and the + hybrid router, Langfuse, Redis Stack, pgvector and agent-sandbox v0.5.0. + `base_domain_name` stays EAS's default (`solutions.ai`): the guide's example value is a + placeholder, and every request below resolves the gateway's name to its address directly. + +**Deviations from stock, named.** + +- **D1, commits.** The rc.1 layer manifest (`configs/repos/repos.agent-toolkit.yaml` lines 16 and + 25) names branches (`release-1.1`, `release-2.0.0`), with the commits in comments on the same + lines. D1 rewrites those two `rev:` values to the commits the comments name (`170eb3a`, + `036b662`), the release notes' component table, before `init`. Nothing else in the file + changes. `h62.py eas-pin` refuses a manifest whose branch or comment differs. +- **D2, weights.** `gemma-4-26b`'s `model_id` in `env/local/models.yaml` (the catalog `init` + copies, the file the guide points to for model settings) becomes + `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized`, so both arms serve the same QAT-trained + weights. This is the same deviation as amendment M1's B1. Every other field of the entry, and + every other entry, is unchanged. EI's download job fetches the weights with `hf download` and no + revision; the revision it fetched is read from the download's metadata and recorded, and the + weight gate checks the files. +- **D3, the guest.** Virtualization itself. + - Both guests share one shape: KVM, `-cpu host`, 96 vCPUs as 1 socket of 48 cores with 2 + threads (crescent's topology), unpinned vCPUs and no hugepages. VM a has 96 GiB, VM b 192 GiB + (B5's stack requests about 163 GiB: gemma 128, Qwen3Guard 16, router 2 and about 17 for the + rest, per the guide's formula, line 84). + - User-mode networking, with each guest's ssh and serve ports forwarded to 127.0.0.1 only. MetalLB's + single-node pool is the node IP /32 (EAS `roles/metallb/tasks/install.yaml` line 200): here + `10.0.2.15`, which does not overlap Calico's `10.233.64.0/18`. + - Ubuntu 24.04 cloud image `release-20261017`, sha256 `04e8f246...6c94`. cloud-init keeps the + image's package state: no upgrade at boot, and unattended-upgrades and the apt timers are off, + so no apt run competes with the installs. + - QEMU 10.2 (`qemu-system-x86` 1:10.2.1+ds-1ubuntu3.3) runs in a container built from + `ubuntu:26.04@sha256:6a06111c...`, with `--device /dev/kvm`, so the host gets no package. QEMU + passes AMX through from 7.0 on, and the host kernel (7.0) supports AMX for KVM guests. +- **D4, the request path.** Intel's agents reach the model through EAS's LiteLLM gateway, with the + content-safety guardrail and the hybrid router (EAT `config.yaml` lines 20-28; the guide's + "Verify the Base Stack" calls `https://litellm./v1/chat/completions`). The + harness reaches B5's workload Service directly instead (below), so B5's turns skip that + gateway, the guardrail's Qwen3Guard check and the router. Each skipped hop is a step every + stock turn takes (the guardrail is a model call), so D4 leans toward B5. The guardrail stays + deployed and holds its CPUs. + +**The harness, shared by both arms, as in H50.** + +- On the host, per arm: the harness's LiteLLM (`v1.83.14-stable.patch.2@sha256:73cbb215...`), + the recording proxy (profile `agent_gemma4`), the banking agent and the MCP sidecar from + blueprint `758d484`. +- Each arm's server is reached through a `kubectl port-forward` of its Service inside its guest, + forwarded to the host. For A1 that is `svc/ovms`. For B5 it is + `gemma-4-26b-kserve-workload-svc:8000`, the Service EI's own LiteLLM bypass uses. +- B5's route therefore skips EAS's Envoy AI gateway, Istio, LiteLLM and the guardrail + (deviation D4). H50 compared engines behind one constant gateway, and with one serving + replica the endpoint picker has one endpoint. EAS's gateway and guardrail are exercised in + the wiring smoke and are not gated. + +**Model and sampling.** Amendment M1's, unchanged. Both arms serve the QAT-trained weights: A1 +the snapped INT4 IR, B5 the bf16 safetensors. Sampling is profile `agent_gemma4` (temperature +1.0, top_p 0.95, top_k 64, max_tokens 1024, thinking off), pinned by the recording proxy for +both arms. B5 serves without `--max-model-len` (the model's 262144), where H50's B1 had 8192; B24's +prompts fit both, and the difference is part of stock B5. + +**Workload, design, statistic and verdict: H50's.** + +- B24 at N=1 under `typical` background load (`bench/load/load.py` on the host's CPUs 24-47 and + 72-95), both arms live in one hold, interleaved pairs. +- Rule L3 (amendment A3) with seed `H62-crescent-2026-10-29`. Each pair's arm order is the seeded + coin, L3's successor to the A B B A order: a pure function of the seed and the pair index, so a + resumed hold draws the same order. +- Each attempt is preceded by the settle gap and the 10 s pre-window. The gates are the balance + gate, at least 16 complete pairs, at most 2 void pairs, and the idle-arm check. The idle-arm + check reads the other arm's serving pod inside its guest (`pairs.py`'s `exec:` cgroup read + through `bench/h62/arm.sh`); the median must stay at or under 0.5 CPU. +- 120 s turn budget, 240 s task cap, and H49b's outcome classes and grader. +- Scorer: `agentbench pairs-report --hyp H62 --arm-a A1 --arm-b B5`, H50's banking scorer + unchanged (`pairs.DESIGNS["H62"]` is `("H49", "L3")`). +- **SUPPORTED** when the paired difference's 95% CI lower bound is at least +30 points and A1's + within-budget success is at least 0.6, with every gate passed. **REFUTED** when the CI's upper + bound is under +30 points. Otherwise **INCONCLUSIVE**, including a row that is not deliverable. +- Reported, never gated: + - each arm's median attempt and turn p95, and the paired J per attempt (RAPL package energy, + which now includes both guests); + - the period-balanced estimate and L2's matched subset; + - the serving covariate, plus each guest's CPU-s/s (`serving.py`'s `vm_cpu`); + - B5's rendered CPU request, cpuset, `nproc` and the ISA its pod sees, per hold; + - the VM tax: A1 in its guest against H50's A1 on the host, and B5 in its guest against H50's + B1 on the host (guest and deployment together), within-budget success and median attempt + (`h62.py vmtax`). + +**Holds, caps and gates.** + +1. **Runner first.** This PR: the runner and its tests, before any guest exists. +2. **Deploy commit.** Every step runs from one clean, detached checkout at + `~/k3dgpu-bench/k3d-gpu-h62`, at an `origin/main` commit at or after this PR's merge, and + with #260 (whose `integrations/eat/eat-check.sh` has the EAS mode the smoke uses). If #260 has + not merged when crescent is ready, a run branch merges #260's head into main, as amendment E3 + did, and the run section names it. VM a gets the same commit (`git archive`). The commit goes + into every record. +3. **Setup phase**, not a hold, inside `flock -w 7200 ~/k3dgpu-bench.lock` (`bench/h62/setup.sh`). + - Host gate. MemAvailable must cover both guests plus 48 GiB (336 GiB; 357 GiB was available + at the feasibility read). `/` must be at most 90% used. Hugging Face, Docker Hub and + registry.k8s.io must answer. It is re-read every 10 min for up to 60 min. + - If the host gate still fails, the two qwen38 serving pairs are stopped under the box owner's + grant (`gpu-serving.sh stop`: `docker stop`, never `rm`, refused while aiperf runs) and setup + runs once more. After the teardown they are started again and checked through Dynamo + `:8100`, as amendment E2 item 7 does; that check gates nothing. + - A1's IR is checked against its manifest. The Hub's file list of the QAT checkpoint at + `f1e06dc5` is fetched for B5's weight gate. + - Guest gate: in each guest, `amx_tile` on 96 of 96 vCPUs and `nproc` 96. A failure means the + row is not deliverable (INCONCLUSIVE). + - Ceilings: VM a's setup (`guest-a.sh`) 45 min, VM b's install (`guest-b.sh`) 120 min, run side + by side. Intel's own estimate is 20-40 min plus 10-15 min of weights; Gate 2's inference-only + install took 863 s (`docs/intel/upstream-eat.md`, test plan row 2). An overrun means + INCONCLUSIVE ("not deliverable"), recorded as a finding. + - NFD gate: VM b's node carries `cpu-cpuid.AMXBF16=true`. + - Weight gate: every LFS file in B5's model directory equals the Hub's sha256 at `f1e06dc5`, + read in B5's pod. + - A failed NFD or weight gate means no hold runs until a dated amendment. +4. **Wiring smoke** (`bench/h62/smoke.sh`, E3-style). It is discarded, disclosed in the run + section, and not one of the 8 holds. + - In VM b, under the lock: `./es_auto_installer.sh validate agent-toolkit`, then + `integrations/eat/eat-check.sh` with `EAT_MODE=eas` through EAS's own gateway on + `gemma-4-26b`: a chat, a get_weather tool call, and a sandbox run that must print 385. + - Then one hold, `H62-smoke`, with the pair-start guard past its end: both serving pods + restart and answer, both warm-ups answer, no pair starts. + - All of it must pass before hold 1. A failure means no hold until a dated amendment. +5. **Holds** (`bench/h62/pairs-hold.sh N` under `bench/eatflow/crescent/hold.sh`). + - At most 40 min each (`LIVE_CAP=2370` inside `timeout -k 30 2400`), until 24 pairs or 8 holds. + - Each hold restarts both serving pods, then needs both Ready with at least 900 s of the hold + left. A hold that misses that is "not deliverable": it counts as one of the 8 and runs no + pair. After two such holds in a row, no further hold runs (INCONCLUSIVE, not deliverable). + - The 600 s pair-start guard applies. + - Hold-start rule (shared rule 1): with load1 over 12 or an aiperf process running, poll every + 2 min for up to 60 min, then run the hold flagged. + - The serving pairs are not stopped (unless the setup fallback above stopped them), and the + serving covariate is recorded. + - For scale: H50's one hold made 24 pairs between 07:16:33Z and 07:37:07Z, so 1 or 2 holds is + the likely count. +6. **Teardown** (`bench/h62/teardown.sh`), after the last hold or any abort. Both guests are + powered off and removed, and their disks, the cloud image and the QEMU image are deleted. The + results stay. +7. **One hold at a time** on crescent (amendment E2 item 4). The guests stay up from setup to + teardown, and other rows run no hold in that window. A hold of another row that does run then + sees the guests as foreign serving load: their containers carry no `k3dgpu` prefix, so its + `serving.py` counts them, and `hold.sh`'s cleanup never removes them. +8. **Records** that can hold a secret (the install logs, which handle the LiteLLM master key and the + sandbox router token) stay on crescent. Every record committed with the run section is first + scanned for those values (F3's secret scan). + +**Risks, named before measuring.** + +- AMX passthrough and the VM tax are unmeasured on crescent. The guest gate catches the first; + the VM tax is described for both arms. It need not be equal: vLLM CPU's per-core OpenMP + threads and OVMS's pool can respond differently to vCPU preemption on unpinned vCPUs, so a gap + between the two descriptions is reported beside the verdict. +- Kubespray rc.1 on one node behind user-mode networking has never run here, and that networking + carries roughly 60 GB of images and weights into VM b. +- VM a reads A1's IR over virtfs (9p). If the cloud image's kernel lacks the 9p modules, + `guest-a.sh` stops at its IR step, and an amendment names the fix before any hold. +- EI has no weights-revision pin, and EAS pulls the vLLM image by tag. The weight gate covers the + first; each hold records the pod's image ID for the second. +- Qwen3Guard deploys first, so `cpu: auto` sizes gemma's balloon to the cores left after it. The + deploy order and B5's rendered request and cpuset are recorded. +- Newer upstream releases not used, so stock pins stay: Kubespray v2.32.0 (stock v2.30.0), vLLM + v0.32.0 (stock v0.28.0) and KServe v0.21.0 (stock v0.18.0, EI `roles/kserve`). For the guests, + QEMU 11.0.3 is packaged only in Ubuntu series newer than crescent's 26.04, whose `-updates` + pocket carries 10.2.1. +- `qemu-system-x86` 1:10.2.1+ds-1ubuntu3.3 is the archive's current `-updates` version. If it is + superseded before the image is built, the build fails and an amendment re-pins. +- Both guests' vCPUs float over all 96 host CPUs, the load CPUs included. VM b is never idle at + the OS level: Kubernetes, Istio, observability and Qwen3Guard run throughout. That background + applies to both arms' attempts alike and is inside every pre-window. + +**Re-run trigger.** + +- `bench/h62/upstream.sh` (`h62.py trigger`) reads both repositories' tags. It fires on an EAS + release tag at v1.2.0 or later (`vX.Y.Z`, not a release candidate) or an EAT release tag at + 2.0.0 or later (`release-X.Y.Z`, its `-hotfix` form, or `vX.Y.Z`). Full support may ship under + a later number than v1.2.x or 2.x, so any later release fires. It is run before setup and + before each hold, and its output goes into the records. +- Fired before hold 1: a dated amendment re-pins B5 to the tagged release, by amendment E1's logic + (EAT stock is what Intel ships now). If the tag's components equal the pinned commits, the + amendment only records that. +- Fired after hold 1: H62's holds finish at the pinned commits, and H62 keeps its verdict. A new + row, H62r, is pre-registered: the same design with B5 at the tagged release (its release notes' + component table). It runs after H62. + +**A-EAS, later, not this row.** k3d-gpu's own EAS path: +`k3d-gpu eas up --agent-toolkit --eas-ref v1.2.0-rc.1` (PR #260) with `gemma-4-26b`, on a k3d-gpu +cluster. Its `pins.json` profile `v1.2.0-rc.1` names the same EAS, EI and EAT commits as B5's D1, +so A-EAS and B5 run the same EAS. It uses a bastion and a bring-your-own kubeconfig, with no +Kubespray, so it can run on crescent's host. It is pre-registered as its own crescent F row after +#248-#251 and #260 merge and F3 passes on the desktop: F3's checks plus `validate agent-toolkit`, +with an optional descriptive B24 against B5. Its expected choices on crescent: `nri-balloons` +(patch 0003: no baked balloons on a single-NUMA, non-hybrid node), `require_amx: true` and +`nfd_enabled: false`. + +**Runner (this PR), with regression tests.** + +- `bench/eatflow/pairs.py`: `DESIGNS["H62"] = ("H49", "L3")` and its order seed. A pod cgroup + may be `exec:`, a command that prints a pod's `cpu.stat` on another machine, and + `pairs.py usage` reads each spec once. +- `bench/eatflow/live.sh`: `ARM=H62` in `LIVE_MODE=pairs` drives two remote arms + (`LIVE_REMOTE_A`, `LIVE_REMOTE_B`): restart, the 900 s readiness rule, records, the + port-forward and the cgroup spec. From the LiteLLM gateways on, the hold is H50's code. The + local arms run unchanged in the other branch. +- `bench/eatflow/crescent/serving.py`: `SERVING_VMS` records the guests' CPU as `vm_cpu`, outside + the serving covariate. +- `bench/h62/`: + - `h62.py`: the pins, the guests' shapes, QEMU command lines and cloud-init, the gates, D1, the + documented edit and D2, the pod picks, the weight-check script, the re-run trigger and the VM + tax. + - `vm.sh`, `qemu.Dockerfile`, `guest-a.sh`, `guest-b.sh`, `setup.sh`, `smoke.sh`, `arm.sh` (with + `arm-A1` and `arm-B5`), `pairs-hold.sh`, `campaign.sh`, `teardown.sh` and `upstream.sh`. +- Tests: `bench/h62/test_h62.py`, `H62Test` in `bench/eatflow/test_pairs.py` and `ServingVmTest` + in `bench/eatflow/crescent/test_crescent.py`. + +**Reproduce (planned).** On crescent, from `~/k3dgpu-bench/k3d-gpu-h62` at the deploy commit: +`bench/h62/upstream.sh`, then `bench/h62/campaign.sh` (setup, smoke, holds, score, teardown). +Records go to `~/k3dgpu-bench/results/H62/`, with the hold records in +`~/k3dgpu-bench/runs/H62-hold/` and the smoke in `~/k3dgpu-bench/smoke/H62/`. They come back +under `bench/eatflow/results/2026-10-29/H62/`. + ### H53 pre-registration: EAT's banking flow on Arc Pro B70 (2026-10-20) Written 2026-10-20, before any H53 measurement and before either H53 arm has run on a GPU. The @@ -4325,6 +4681,227 @@ interval), REFUTED (0.3), SUPPORTED (0.1). smoke, then holds 1-6 (and 7-8 if needed), then serving restored and verified. The window is at most eight 40-min holds with both pairs stopped; `gpu-state.txt` says so throughout. +### H53 amendment R1: the solo runner fix, setup, wiring smokes and the GPU window (2026-10-30) + +Recorded 2026-10-30 (UTC; 2026-10-29 in California), before any H53 hold or smoke. Mechanics +only. No hypothesis, arm, model, sampling, prediction, statistic, gate, threshold, seed or order +changes, and no cap rises. Like E1 to E4, it is pre-registered by its push time. + +State on crescent at 00:01Z: the only H53 record is `results/H53/prehold/fetch.log` (item 8). +`runs/campaign.log` and `runs/holds.log` have no H53 entry, no H53 process, cluster or lock +exists, the bench lock is free and both SGLang pairs serve. + +Disclosed: it is written with H50's, H50o's, H51's and H52b's results known. None of them runs +on a B70 or serves Qwen3.8-27B. + +1. **Deploy commit.** E2 item 2 runs each hold from `origin/main` at or after E2's merge, and E4 + item 2 runs E4's setup from a commit at or after E4's merge. H53 runs from one pushed commit on + branch `claude/h53-run`: `origin/main` `74ff63ef` (k3d-gpu 0.1.80, after E4's merge in + `012b4e95`) plus this amendment, item 2's fix, item 3's guard, item 5's futility helper and the + window script of item 5. It is checked out clean and detached at `~/k3dgpu-bench/k3d-gpu-h53`, + so a redeploy of the shared checkout cannot move H53's runner between holds. Its commit goes + into every hold's `commit.txt` and the run section. AG is `agent manifest` at this commit. +2. **The fix: `stage_path` was undefined on the solo path.** Since #157, `live.sh` defined + `stage_path` only inside its pairs branch, and the solo branch runs before it. AG's + `stage_path "${TGT_DIR}" "${TGT}" 5000` would have failed with "command not found". `live.sh` + runs without `-e`, so it would have gone on and deployed OVMS over an empty hostPath, and + hold 1 would have ended NOT DELIVERABLE because of the harness, not the arm. The function now + sits at top level, unchanged. The pairs path calls the same function. B2 mounts its weights + read-only (`LIVE_MOUNT_B=1`) and never called it. Regression test: + `H53Test.test_stage_path_is_defined_before_the_solo_path_calls_it` in + `bench/eatflow/test_pairs.py`. It fails on `74ff63ef`. +3. **A guard for the smokes.** `live.sh`'s solo branch passes `LIVE_PAIRS_GUARD`, when set, as + `agentbench solo --guard-s`. The scored holds leave it unset (`solo-hold.sh` does not set it), + so their guard stays agentbench's own. A smoke sets 99999 s: no task starts, and only the + unscored warm-up runs. +4. **Wiring smokes, inside the window, before hold 1.** No H53 arm has run on k3d. The + pre-registration's B2 boot-ladder smoke runs, and so does one for AG. Each is + `prehold.sh smoke [RUNGS]`: `hold.sh H53-smoke- 2400` running `live.sh H53` with + `solo-hold.sh`'s environment, except `LIVE_PAIRS_GUARD=99999`, `LIVE_HOLD` 1 for AG and 2 for + B2, and output under `~/k3dgpu-bench/smoke/H53//`. Each is discarded, disclosed in the + run section, and is not one of the eight holds. + - AG smoke (`H53-smoke-AG`). Pass: OVMS is ready on the GPU (`--target_device=GPU` in + `agent.yaml`), its pod holds exactly one free card (`live.sh`'s own check), and the warm-up + answers. If it fails for a reason the harness or k3d causes, no hold runs until a further + dated amendment fixes it. If OVMS itself does not load or answer, nothing changes, and + hold 1 runs as registered. + - B2 boot ladder (`H53-smoke-B2-a`, then one smoke per further rung, named by its letters, + such as `-b` or `-bd`). This is the pre-registration's ladder as written. Rung (a) is stock + with item 1's mounts dropped. A rung's smoke runs only when the boot before it fails with + that rung's error, and every rung taken is kept for every B2 hold (`LIVE_B2_LADDER`, the + `RUNGS` argument of `holds`). Pass: the server is ready, its pod holds the four free cards, + and the warm-up answers. If B2 does not answer after (d), H53 is INCONCLUSIVE (not + deliverable), no hold runs, and the failure is the finding. + - Early stop. A smoke may stop once its server is ready and the warm-up has answered, or once + its server has failed at least twice with the same error. Before stopping it, + `prehold.sh podlog ` saves the pods, events and server logs (current and previous). + Then a TERM goes to that smoke's `hold.sh`, whose cleanup deletes the cluster. +5. **The window** (`bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh`), in this order: + `setup` (item 7), `verify` (item 8) and `frontdoor pre` before the window; `stop`; the smokes; + `holds 1 6 RUNGS`; `start`. + - `stop` saves `docker inspect` of both pairs into `prehold/` first. Then it runs + `gpu-serving.sh stop` for `qwen38-a-hr` and `qwen38-b-hr` (`docker stop`, never `rm`; it + refuses while aiperf runs). + - Before every smoke and hold, the window must be open, both pairs must be stopped, and every + card must hold under 2 GiB of VRAM (Prometheus `xe_vram_used_bytes`, polled for up to + 2 min). Else the step does not run. The serving pairs hold about 16 GiB of each card. + - `holds FIRST LAST RUNGS` runs the scored holds in order through `solo-hold.sh N ARM`, with + `LIVE_GPU_FREE` set to all four cards. After hold 6, holds 7 and 8 run only when `agentbench + solo-report` (on a copy of the solo directory) counts fewer than three blocks. This is the + pre-registered rule. Then it restores serving. + - A hold that does not exit 0 counts as used, and `holds` stops there with the window open. + The hold's records are read before the next hold. A failure the harness or k3d causes takes a + further dated amendment before any further hold. A failure of the arm itself (it does not + load, does not answer, or stops answering) is that arm's result, and the holds go on in + order from the next one. + - Futility stop. After every hold, `solo.alive_blocks` lists the blocks (1 to 4) that can + still count: a block is out when one of its holds has run with fewer than 8 complete + attempts, or when both have run and it has fewer than 8 complete pairs. When fewer than + three remain, no further hold runs and serving is restored. The verdict is then INCONCLUSIVE + by the pre-registered rule (fewer than three blocks count), whatever later holds would show. + The rule reads only block counts, never d or a success rate, and it shortens the outage of + the serving pairs. Tests: `AliveBlocksTest` in `bench/eatflow/test_solo.py`. + - `start` (and the end of `holds`) runs `gpu-serving.sh start` for both pairs. That waits for + `/health`, `/v1/models` and a 16-token chat at each pair's own port. It then compares each + pair's image, command, env, devices, mounts and ports with the saved inspect, and runs E2 + item 7's front-door check through `:8100`. The front-door check gates nothing. + - A deadman runs from `stop`. If no window step is alive and none has run for 45 min, it + takes the bench lock and restores both pairs, so a dead session cannot leave serving down. + It stays one window (E2 item 4): smokes and holds 1-8 run with both pairs stopped, and + serving comes back after the last. +6. **Placement records.** `live.sh` already stops a hold whose pod holds a card outside + `LIVE_GPU_FREE` or the wrong number of cards. For every hold, the window script also saves, + from Prometheus, each card's VRAM and engine use over the hold (`hold/xe-prom.json`). These + are reported and gate nothing. +7. **Tools (crescent amendment E4).** E4 merged before this push, and H53 is one of its rows. So + `prehold.sh setup` runs `bench/eatflow/crescent/setup.sh` once from H53's checkout, under the + bench lock, before the first smoke. Its output, the checksum lines and the four versions, goes + to `prehold/setup.txt`. Every smoke and hold then runs on k3d v5.9.0, kubectl v1.37.2, helm + v4.3.1 and uv 0.13.6 in `~/k3dgpu-bench/bin`. E4's render check found B2's staged objects + byte-identical under helm v4.1.1 and v4.3.1. setup.sh also re-extracts `eat-pr21` from + `c86c42b`, the tree it already holds, and re-checks the banking venv's packages. +8. **Models.** `models.sh qwen38` ran under the bench lock from 21:44Z to 21:55Z on 2026-10-29, + from the main-tracking checkout at `6d5cab69`, before this amendment existed. `models.sh` and + `listing.sh` are unchanged from `6d5cab69` to the deploy commit. It downloaded both models at + their pinned revisions, checked every LFS file against the Hub's sha256 (58 files, all `ok`, + rc 0) and wrote B2's listing, which `live.sh` checks at every B2 hold. Its log is + `prehold/fetch.log`, the record `prehold.sh fetch` writes. It measures nothing. Before the + window, `prehold.sh verify` checks both models against the Hub again, and B2's listing against + setup's (`prehold/verify.txt`). +9. **Records.** The results come back under the pre-registration's date, + `bench/eatflow/results/2026-10-20/H53/`. `solo-hold.sh` writes `pairs/` and `hold/`, and + `hold.sh` writes `runs/H53-hold-/`. The run also keeps `prehold/`, `window/` and + `smoke//`. The score is `agentbench solo-report bench/eatflow/results/2026-10-20/H53/pairs + --hyp H53 --arm-a AG --arm-b B2`, at the deploy commit. + +### H53 run (Arc Pro B70, 2026-10-30) + +**Reproduce:** + +```text +# provenance +date: 2026-10-30T00:13:39Z to 00:38:03Z; GPU window 00:14:50Z (both pairs stopped by 00:15:06Z) + to 00:38:03Z (both answering at 00:37:43Z); no scored hold ran +hypothesis: H53 +host cpu: Intel(R) Xeon(R) 6741P (Granite Rapids), 48 cores / 96 threads, 503 GiB RAM +host kernel: 7.0.0-31-generic; Docker 29.8.0 +accelerators: 4x Intel Arc Pro B70 (PCI 8086:e223, driver xe) at 0000:12:00.0, 0000:48:00.0, + 0000:7c:00.0 and 0000:96:00.0, all four free for the window; k3d-gpu up: intel=4 amx=1 +k3d-gpu commit: 1f39488b (branch claude/h53-run: origin/main 74ff63ef, k3d-gpu 0.1.80, plus amendment R1), + checked out clean and detached at ~/k3dgpu-bench/k3d-gpu-h53 +node image: cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c; + k3d v5.9.0, kubectl v1.37.2, helm v4.3.1, uv 0.13.6 (amendment E4's setup.sh, run at 1f39488b + under the bench lock, every checksum OK: prehold/setup.txt) +model: AG OpenVINO/Qwen3.8-27B-int4-ov @ 7b20ea8c; B2 Qwen/Qwen3.8-27B @ 1d4bf0f2 (bf16, mounted + read-only, listing equal to setup's); every LFS file of both equal to the Hub's sha256 at + download (prehold/fetch.log) and again before the window (prehold/verify.txt) +runtime: AG `agent manifest` (auto device, default draft rule): openvino/model_server@sha256:a928903f... + (2026.4.1), --target_device=GPU, gpu.intel.com/xe 1, tool_parser qwen3coder, no reasoning + parser, no draft; B2 EAT release-1.1.0 XPU path (the c86c42b tree) core/helm-charts/vllm + xpu-values.yaml, LLM_MODEL_ID=Qwen/Qwen3.6-27B, vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a..., + tensor_parallel_size 4, gpu.intel.com/xe 4; LiteLLM v1.83.14-stable.patch.2@sha256:73cbb215...; + blueprint 758d484, ENABLE_MEMORY=false; sampling agent_qwen38 (temperature 0.7, top_p 0.8, + top_k 20, presence_penalty 1.5, max_tokens 1024, thinking off) +load: typical, in the AG smoke only (the B2 smokes ended before load starts) +runs per arm: 0 of the 8 permitted holds; 3 wiring smokes, discarded +``` + +Commands, on crescent, from the H53 checkout: +`bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh` steps `setup`, `verify`, `frontdoor pre`, +`stop`, `smoke AG`, `smoke B2`, `smoke B2 b`, then `start`. No `holds` step ran, so there is no +`solo-report` score. Records: [bench/eatflow/results/2026-10-20/H53/](bench/eatflow/results/2026-10-20/H53/) +(`prehold/`, `window/` with `events.log` and `xe-prom.json`, `smoke/` with each smoke's records and pod +logs, `runs/` with each smoke's `hold.sh` records). + +**Before the window, all at `1f39488b`:** setup.sh installed E4's four tools, every checksum OK +(00:13:39-00:13:42Z). The re-verify read every LFS file of both models equal to the Hub's sha256, and B2's +listing equal to setup's (rc 0 for each). Through Dynamo `:8100`, each serving pair answered "OK" and its +`num_requests_total` rose by one: :30000 161 to 162, :30001 57 to 58 (`prehold/frontdoor-pre.txt`, 00:14:45Z). + +**The window.** `stop` saved `docker inspect` of both pairs, then stopped `qwen38-a-hr` (00:15:00Z) and +`qwen38-b-hr` (00:15:06Z) with `docker stop`. The cards read 16.14, 15.43, 16.18 and 15.47 GiB used before +the stop and 0.0-0.17 GiB after it. + +- **AG smoke** (`H53-smoke-AG`, 00:15:13-00:20:00Z): passed. `agent manifest` chose the GPU + (`--target_device=GPU`, `gpu.intel.com/xe: 1`; its fit: one GPU, context 32768 x 6 sequences). OVMS loaded the + IR as a VLM continuous-batching servable and was ready 51 s after its deploy (241 s into the smoke, after + `up` and the 16 GB IR copy). Its pod held one card, 0000:7c:00.0 (`gpu-arms.txt`). The warm-up answered in + 5.5 s. Prometheus read 14.81 GiB peak on that card and 0.00 on the other three. +- **B2 rung (a)** (`H53-smoke-B2-a`, 00:20:14-00:25:26Z): failed. The container exits at once with + `/opt/intel/oneapi/setvars.sh: No such file or directory`, which is rung (b)'s error, as the + pre-registration had read from the image (item 2). Stopped early after two restarts. +- **B2 rung (b)** (`H53-smoke-B2-b`, 00:25:35-00:36:28Z): failed. Without the `setvars.sh` source vLLM + v0.28.0 starts, and its four XPU workers then fail in their first collective + (`xpu_worker.init_device`, `torch.distributed.all_reduce`): + `RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open + device directory`. The engine core then exits with "Engine core initialization failed". It is the same + error at 00:33:38Z and 00:35:27Z (two saved attempt logs), and the pod restarted three times. The pod held + all four cards and touched each (0.51-0.84 GiB peak) before the failure. +- **Why rung (b) fails** (`smoke/H53-smoke-B2-b/hold/devdri-diag.txt`, read while the pod ran): the image's + oneCCL (`/opt/venv/lib/libccl.so.1`) opens `/dev/dri/by-path/`. The host has that directory. The k3d + node container does not: it gets `/dev/dri/cardN` and `renderDN` as devices, which carry no `by-path` + symlinks. The Intel GPU plugin mounts the node's `/dev/dri`, so the pod's `/dev/dri` has the four card + and render nodes and no `by-path`. The gap is in k3d-gpu's device exposure, not in EAT. This run shows it + for B2's oneCCL settings (`CCL_ZE_IPC_EXCHANGE=pidfd`, from the chart); another multi-card oneCCL job that + opens the same directory would meet it too. A one-card arm makes no collective, which is why AG is + unaffected. +- **The ladder ends at (b).** Rung (c) drops `--disable-log-requests`, which B2's command does not carry. + Rung (d) applies only to a KV-cache failure. Neither applies to this error, and the pre-registration + allows no other change. So B2 does not answer, and the pre-registration's rule applies: "If B2 does not + answer after (d), H53 is INCONCLUSIVE (not deliverable) and the failure is the finding" (R1 item 4: no + hold runs). + +`start` restarted both pairs at 00:36:38Z. `gpu-serving.sh` read `/health` 200, `/v1/models` and a 16-token +chat from `qwen38-b-hr` at 00:37:40Z and `qwen38-a-hr` at 00:37:43Z. Each pair's image, command, env, +devices, mounts and ports equal its inspect from before the stop. Through `:8100`, each pair answered "OK" +and its `num_requests_total` rose: :30000 2 to 3 (00:37:52Z), :30001 3 to 4 (00:38:03Z) +(`prehold/frontdoor-post.txt`). The serving outage was 22.7 min (`qwen38-a-hr`, 00:15:00-00:37:43Z) and 22.6 min +(`qwen38-b-hr`, 00:15:06-00:37:40Z). No `xe` kernel message was logged during the window (`runs/*/dmesg-*.txt`). Every smoke's +cleanup deleted its cluster, the deadman exited with the window, and the bench lock was free afterwards. + +**Predictions, read as written:** + +- AG >= 0.70 (0.6): not measured. No hold ran. +- B2 boots after at most rung (b) (0.6) and answers (0.75): **missed**. B2 did not boot after rung (b), + and it did not answer. +- B2 >= 0.50 (0.5), block mean AG - B2 in [-0.15, +0.30] (0.5), three blocks count (0.7): not measured. +- Verdict INCONCLUSIVE (0.6): matched, but by not-deliverable rather than by a wide three-block interval. + +**Verdict: INCONCLUSIVE (not deliverable).** EAT release-1.1.0's XPU path for a 27B Qwen at TP 4 did not +start in a k3d-gpu cluster on this host. It needed rung (b) (its command sources a `setvars.sh` the v0.28.0 +image lacks), and then it failed on k3d-gpu's device exposure: the node has no `/dev/dri/by-path`, which +oneCCL needs for collectives across cards. k3d-gpu's one-card choice, AG, deployed and answered. No block +ran, so H53 says nothing about the stack-vs-stack claim. + +Scope: one host, one node image (0.1.61) and one k3d-gpu commit (`1f39488b`). The by-path gap is k3d-gpu's, +not EAT's. On a host where pods see `/dev/dri/by-path`, B2 might pass rung (b), which H53 does not test. + +Disclosed: amendment R1 was pushed at 00:13:25Z, before the first H53 step (`setup`, 00:13:39Z), and was not +merged before the run. Its futility rule and per-hold placement records did not bind, since no hold ran. +The model download (R1 item 8) ran on 2026-10-29 before R1 existed and measures nothing. The per-card +VRAM and engine record of the window (`window/xe-prom.json`) was read from Prometheus after the window, +read-only. + ### H54 pre-registration: Gemma 4 26B-A4B QAT and Qwen3.8 microbenches on crescent (2026-10-20) Written 2026-10-20, before any H54a, H54q, H54b, H54c or H54d measurement. Nothing here is a @@ -5203,6 +5780,36 @@ H54d is recorded INCONCLUSIVE (not run). Running it would need a dated amendment without B1g's score (or a working B1g), a runner amendment (the pairs runner has no llama.cpp arm A), and the tool-call wiring smoke. Record: `bench/h54/results/2026-10-21/H54d/not-run.txt`. +### H54d: re-checked, still not run (2026-10-29) + +Recorded 2026-10-29 (2026-10-30T00:55Z), after the H54q run, note N2, amendment E1 and the H50 +run. No hold used, no cluster created, nothing on crescent changed. No prediction, rule, cap or +verdict changes: H54d stays INCONCLUSIVE (not run). The 2026-10-21 record's third reason (H54q had +not run) no longer applies. Its other reasons stand, and two facts are new. + +1. **A* is still undefined.** H54a is closed: its 3 holds are spent and R3's one carry is used. Its + rule picks the fastest arm among those whose GSM8K score is at least B1g's minus 5 of 50. B1g + has no score, so `h54.py score-a` chose none. H54q's reference R now has 200 items, but H54a's + rule does not read R. Gating on R would be a new selection rule. +2. **B1g still stalls.** Amendment E1 moved H50, H50o and H51 to EAT release-1.1.0 (vLLM CPU + v0.28.0). It kept H54a's and H54d's B1g at v0.24.0, both closed. Note N2 places B1g's stall on + vllm-project/vllm#45844, which v0.24.0 lacks. All three v0.24.0 Gemma engines measured on + crescent stalled within their first 690-1,540 decode steps. Under H50's procedure a stalled pod + trips the idle-arm check and the quiesce wait, and a stall in the warm-up stops the hold before + any pair (E1). A hold would measure the gates, not the arms. +3. **New: re-opening it now would not be a pre-registration.** A run needs a dated amendment that + defines A* and moves B off v0.24.0. Written now, it would come after the H50 run on the same + flow, where E1's B1 (vLLM CPU v0.28.0) passed 21 of 24 B24 tasks within budget and A1 19 of 24 + (REFUTED). That is the moved B's outcome, already measured. A* would also be picked after + H54a's decode rates and H54q's 200-item scores are known. So no amendment is written here. +4. **New: one hold left.** Amendment Q1 moved 2 of H54d's 3 holds to H54q. The H54 campaign has + used 9 of its 10 holds (H54a 3, H54q 3, H54b 2, H54c 1); H54d holds the last one. +5. **The runner still has no llama.cpp arm A.** `bench/eatflow/pairs.py design H54d` exits with + "not a pairs hypothesis", and `live.sh`'s pairs mode deploys A1 (OVMS) as arm A in every + design it knows. + +Record: `bench/h54/results/2026-10-29/H54d/not-run.txt`. + ### H54q run (Xeon 6741P, 2026-10-29) **Reproduce:** diff --git a/bench/eatflow/crescent/serving.py b/bench/eatflow/crescent/serving.py index cd61ab74..b086dc67 100755 --- a/bench/eatflow/crescent/serving.py +++ b/bench/eatflow/crescent/serving.py @@ -7,7 +7,12 @@ Slot mode (H56, SERVING_SLOT=i with SERVING_CPUS = the slice): only slot i's containers are this hold's, so a sibling hold's cluster and load count as foreign; each row also carries own_busy_pct / other_busy_pct (/proc/stat over the slice and the rest of the -online CPUs) and own_mhz (mean scaling_cur_freq over the slice).""" +online CPUs) and own_mhz (mean scaling_cur_freq over the slice). + +VM mode (H62, SERVING_VMS = comma-separated container names): those containers run this +campaign's KVM guests, so each row records their CPU-s/s as vm_cpu and leaves them out of cpu +and serving_cpu. Their names carry no k3dgpu prefix, so another row's hold counts them as +foreign serving load.""" import json import os import signal @@ -40,6 +45,19 @@ def _slot(): SLOT = _slot() + + +def vms_from(value): + """SERVING_VMS: the container names of this campaign's guests (H62), or ().""" + return tuple(v.strip() for v in value.split(",") if v.strip()) + + +def split_vms(cpu, vms): + """(serving cpu, vm cpu): the readings of the containers named in vms move to the second.""" + return ({n: v for n, v in cpu.items() if n not in vms}, {n: v for n, v in cpu.items() if n in vms}) + + +VMS = vms_from(os.environ.get("SERVING_VMS", "")) stop = False @@ -113,8 +131,11 @@ def main(out): for n in cs if cur.get(n) is not None and prev.get(n) is not None} with open("/proc/loadavg") as la: load1 = float(la.read().split()[0]) + cpu, vm_cpu = split_vms(cpu, VMS) row = {"ts": round(t1, 3), "seconds": round(t1 - t0, 3), "load1": load1, "cpu": cpu, "serving_cpu": round(sum(cpu.values()), 3), "aiperf_procs": aiperf()} + if VMS: + row["vm_cpu"] = vm_cpu if SLOT is not None: st1 = load._read("/proc/stat") row.update(slot=SLOT, **slot_fields(st, st1, own)) diff --git a/bench/eatflow/crescent/test_crescent.py b/bench/eatflow/crescent/test_crescent.py index 57e4cf5a..69ae73e6 100644 --- a/bench/eatflow/crescent/test_crescent.py +++ b/bench/eatflow/crescent/test_crescent.py @@ -55,6 +55,34 @@ def test_slot_fields(self): shutil.rmtree(tmp) +class ServingVmTest(unittest.TestCase): + """H62 (EXPERIMENTS.md "H62 pre-registration"): the two KVM guests are this campaign's arms. + SERVING_VMS names their containers; their CPU is recorded as vm_cpu and kept out of the + serving covariate. Without it, every row reads as before.""" + + CPU = {"qwen38-a-hr": 1.25, "qwen38-b-hr": 1.0, "h62-vm-a": 3.5, "h62-vm-b": 2.0} + + def test_vms_split_out_of_the_serving_covariate(self): + serving_cpu, vm_cpu = serving.split_vms(self.CPU, ("h62-vm-a", "h62-vm-b")) + self.assertEqual(serving_cpu, {"qwen38-a-hr": 1.25, "qwen38-b-hr": 1.0}) + self.assertEqual(vm_cpu, {"h62-vm-a": 3.5, "h62-vm-b": 2.0}) + + def test_without_vms_nothing_moves(self): + self.assertEqual(serving.split_vms(self.CPU, ()), (self.CPU, {})) + self.assertEqual(serving.vms_from(""), ()) + self.assertEqual(serving.vms_from("h62-vm-a, h62-vm-b,"), ("h62-vm-a", "h62-vm-b")) + + def test_a_vm_missing_from_the_reading_is_not_invented(self): + serving_cpu, vm_cpu = serving.split_vms({"qwen38-a-hr": 1.0}, ("h62-vm-a",)) + self.assertEqual((serving_cpu, vm_cpu), ({"qwen38-a-hr": 1.0}, {})) + + def test_the_guests_are_not_this_campaigns_by_name(self): + # Another row's hold counts the guests as foreign serving load: their names carry no + # k3dgpu prefix, and hold.sh's cleanup (k3dgpu-e-*, k3dgpu-a-*) never removes them. + for name in ("h62-vm-a", "h62-vm-b"): + self.assertFalse(serving.is_mine(name), name) + + class LockwaitTest(unittest.TestCase): def run_log(self, text, waits=()): import lockwait diff --git a/bench/eatflow/live.sh b/bench/eatflow/live.sh index 28a1dba7..e8770887 100755 --- a/bench/eatflow/live.sh +++ b/bench/eatflow/live.sh @@ -43,6 +43,7 @@ # solo: ARM = H53, one arm (LIVE_SOLO) per hold on the Arc Pro B70s (bench/eatflow/solo.py); # pairs: ARM = H47, H49, H47b, H49b, H50 or H50o, both arms live in cluster k3dgpu-e-, one hold # of interleaved pairs into LIVE_PAIRS_DIR (LIVE_HOLD = this hold's number); +# ARM = H62: both servers in KVM guests, driven by LIVE_REMOTE_A and LIVE_REMOTE_B (bench/h62); # code: H43 (amendment W1), the coding agent on C12 with `k3d-gpu sandbox up`, one # block per level in LIVE_CODE_LEVELS (default "typical heavy heavy typical") into # LIVE_CODE_DIR/h43-- (shared by an arm's holds, so a level's halves resume); @@ -170,6 +171,7 @@ t0=$(date +%s) el() { echo $(( $(date +%s) - t0 )); } stamp() { echo "== $1 $(date +%T) (+$(el)s)" | tee -a "${OUT}/live.log"; } pids=() +REMOTE=() # H62: the remote arms' control scripts (their port-forwards stop with the hold) # DevUI bearer token for this hold only (the agent and the harness read DEVUI_AUTH_TOKEN). DEVUI_AUTH_TOKEN=$(python3 -c "import secrets; print(secrets.token_hex(16))") export DEVUI_AUTH_TOKEN @@ -180,6 +182,7 @@ cleanup() { for p in "${pids[@]}"; do kill "${p}" 2>/dev/null; done [ -f "${OUT}/mcp.pid" ] && kill "$(cat "${OUT}/mcp.pid")" 2>/dev/null for f in "${OUT}"/agent-*.pid; do [ -f "${f}" ] && kill "$(cat "${f}")" 2>/dev/null; done + for r in "${REMOTE[@]}"; do "${r}" unforward >/dev/null 2>&1; done docker rm -f "${PFX}-litellm" "${PFX}-litellm-a" "${PFX}-litellm-b" "${PFX}-${lc}-srv" "${PFX}-rapl" \ "${PFX}-redis" >/dev/null 2>&1 k3d cluster delete "${K3D_GPU_CLUSTER}" >/dev/null 2>&1 @@ -280,6 +283,19 @@ stage() { # stage docker exec "${n}" chown -R "$3:$3" /var/apb-models done } +# stage_path : copy into every node, chown that +# path only. Top level, so the solo (H53) and pairs paths both have it (H53 amendment R1). +stage_path() { + local n + for n in $(nodes); do + docker exec "${n}" mkdir -p "/var/apb-models/$2" /var/apb-models/.ov_cache + docker cp -L "${M}/$1/." "${n}:/var/apb-models/$2/" + docker exec "${n}" rm -rf "/var/apb-models/$2/.cache" "/var/apb-models/$2/.git" + docker exec "${n}" chown -R "$3:$3" "/var/apb-models/$2" + docker exec "${n}" chown 5000:5000 /var/apb-models/.ov_cache + docker exec "${n}" chmod 1777 /var/apb-models + done +} # cpusets: H56's cpuset.txt, " " lines that crescent/h56.py checks. cpusets() { local n c @@ -316,6 +332,8 @@ if [ "${MODE}" = solo ]; then # AG (k3d-gpu's auto device and draft: OVMS on one card) or B2 (EAT PR #21's vLLM XPU path, # its xpu-qwen36-27b profile on these weights, LIVE_B2_TP cards). LIVE_GPU_FREE: the PCI # addresses of the cards the hold may use; every other card is held by a placeholder pod. + # LIVE_PAIRS_GUARD: the task-start guard, s (unset: agentbench solo's own; a wiring smoke + # sets it past the hold, so only the warm-up runs; H53 amendment R1). PD=${LIVE_PAIRS_DIR:?} HOLD=${LIVE_HOLD:?} SOLO=${LIVE_SOLO:?LIVE_SOLO is AG or B2} read -r FAMILY RULE <<< "$(python3 "${HERE}/pairs.py" design "${ARM}")" [ "${ARM}" = H53 ] && [ "${RULE:-}" = L3 ] || { echo "solo: ARM must be H53 (a rule-L3 row)"; exit 2; } @@ -428,7 +446,8 @@ if [ "${MODE}" = solo ]; then ${AB} solo --hyp "${ARM}" --hold "${HOLD}" --arm "${SOLO}" --out "${PD}" --deadline "${deadline}" --tasks B24 \ --agent http://127.0.0.1:8090 --upstream "http://127.0.0.1:${P_LLM}" --proxy-port "${P_PROXY}" \ --restart "${OUT}/agent.sh" --pod "${cg}" --mcp-url "http://127.0.0.1:${P_MCP}/mcp" --reset "${OUT}/reset-mcp.sh" \ - --sampling "${SAMPLING}" --baseline "${PD}/baseline-hold${HOLD}.jsonl" --level "${LEVEL}" 2>&1 | tee -a "${OUT}/live.log" + --sampling "${SAMPLING}" --baseline "${PD}/baseline-hold${HOLD}.jsonl" --level "${LEVEL}" \ + ${LIVE_PAIRS_GUARD:+--guard-s "${LIVE_PAIRS_GUARD}"} 2>&1 | tee -a "${OUT}/live.log" stamp finished exit 0 fi @@ -449,6 +468,7 @@ if [ "${MODE}" = pairs ]; then # What to deploy: the family's arms, or the row's own B arm (H50o: B3). DESIGN=${FAMILY} [ "${ARM}" = H50o ] && DESIGN=H50o + [ "${ARM}" = H62 ] && DESIGN=H62 [ "${ARM}" = H50 ] && KV_GIB=${LIVE_EAT_KV_GIB:-40} # EAT's shipped KV cache (40 GiB) mkdir -p "${PD}" PD=$(cd "${PD}" && pwd) @@ -456,23 +476,57 @@ if [ "${MODE}" = pairs ]; then H47) A=A1 B=A4 ;; H49) A=A1 B=B1; export K3D_GPU_NODE_LABELS=ei-infra-eligible=true,ei-inference-eligible=true ;; H50o) A=A1 B=B3 ;; + H62) A=A1 B=B5 ;; esac + if [ "${DESIGN}" = H62 ]; then + # H62 ("H62 pre-registration"): each server runs in its own KVM guest on this host + # (bench/h62); nothing is deployed here. LIVE_REMOTE_A and LIVE_REMOTE_B name each arm's + # control script (bench/h62/arm.sh ): restart its serving pod, record it, forward + # its Service to a host port and name its pod cgroup for the idle-arm check. From the + # LiteLLM gateways on, the hold is H50's. The local arms below run unchanged in the else + # branch. + RA=${LIVE_REMOTE_A:?H62 needs LIVE_REMOTE_A} RB=${LIVE_REMOTE_B:?H62 needs LIVE_REMOTE_B} + REMOTE=("${RA}" "${RB}") + stamp "restart both servers" + left=$(( CAP - $(el) - 900 )) # ready with at least 900 s of the hold left + "${RA}" restart "${left}" > "${OUT}/restart-a.log" 2>&1 & + ra=$! + "${RB}" restart "${left}" > "${OUT}/restart-b.log" 2>&1 & + rb=$! + wait "${ra}"; xa=$? + wait "${rb}"; xb=$? + if [ "${xa}" != 0 ] || [ "${xb}" != 0 ]; then + "${RA}" logs "${OUT}/remote-a" > /dev/null 2>&1 + "${RB}" logs "${OUT}/remote-b" > /dev/null 2>&1 + if [ "${xa}" = 3 ] || [ "${xb}" = 3 ]; then + echo "NOT DELIVERABLE: not ready with 900 s of the hold left (restart A ${xa}, B ${xb})" | tee -a "${OUT}/live.log" + exit 3 + fi + echo "restart failed (A ${xa}, B ${xb})" | tee -a "${OUT}/live.log" + exit 1 + fi + stamp ready + "${RA}" records "${OUT}/remote-a" > "${OUT}/records-a.log" 2>&1 || { echo "A records failed"; exit 1; } + "${RB}" records "${OUT}/remote-b" > "${OUT}/records-b.log" 2>&1 || { echo "B records failed"; exit 1; } + "${RA}" forward >> "${OUT}/live.log" 2>&1 || { echo "A forward failed"; exit 1; } + "${RB}" forward >> "${OUT}/live.log" 2>&1 || { echo "B forward failed"; exit 1; } + P_SRV=$("${RA}" port) BPORT=$("${RB}" port) API_B=/v1 + cg_a=$("${RA}" cgroup) || { echo "no cgroup for the A pod"; exit 1; } + cg_b=$("${RB}" cgroup) || { echo "no cgroup for the B pod"; exit 1; } + echo "pod cgroups: ${A} ${cg_a}; ${B} ${cg_b}" | tee -a "${OUT}/live.log" + python3 "${HERE}/pairs.py" usage "${cg_a}" "${cg_b}" >> "${OUT}/live.log" 2>&1 || { echo "pod cpu.stat unreadable"; exit 1; } + wait_http "http://127.0.0.1:${P_SRV}/v1/models" 120 || { echo "A server not answering"; exit 3; } + curl -fsS "http://127.0.0.1:${P_SRV}/v1/models" > "${OUT}/models-a.json" + wait_http "http://127.0.0.1:${BPORT}${API_B}/models" 300 || { echo "B server not answering"; exit 3; } + curl -fsS "http://127.0.0.1:${BPORT}${API_B}/models" > "${OUT}/models-b.json" + SERVED_A=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['data'][0]['id'])" "${OUT}/models-a.json") + SERVED_B=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['data'][0]['id'])" "${OUT}/models-b.json") + else # The B arm's server (every design but H47's one pod): namespace, Deployment, Service, # service port and OpenAI path. B_NS=default B_DEP=${B_REL} B_SVC=${B_REL}-service B_SPORT=80 B_API=/v1 [ "${DESIGN}" = H50o ] && B_DEP=ovms-model-server B_SVC=ovms-model-server B_SPORT=8080 B_API=/v3 DRAFT_DIR=OpenVINO-Qwen3-0.6B-int4-ov # where OVMS --pull puts the draft, inside the target dir - stage_path() { # stage_path : chown that path only - local n - for n in $(nodes); do - docker exec "${n}" mkdir -p "/var/apb-models/$2" /var/apb-models/.ov_cache - docker cp -L "${M}/$1/." "${n}:/var/apb-models/$2/" - docker exec "${n}" rm -rf "/var/apb-models/$2/.cache" "/var/apb-models/$2/.git" - docker exec "${n}" chown -R "$3:$3" "/var/apb-models/$2" - docker exec "${n}" chown 5000:5000 /var/apb-models/.ov_cache - docker exec "${n}" chmod 1777 /var/apb-models - done - } cluster_up stage_path "${TGT_DIR}" "${TGT}" 5000 [ "${A_DRAFT}" = off ] || stage_path Qwen3-0.6B-int4-ov "${TGT}/${DRAFT_DIR}" 5000 @@ -590,6 +644,7 @@ if [ "${MODE}" = pairs ]; then curl -fsS "http://127.0.0.1:${BPORT}${API_B}/models" > "${OUT}/models-b.json" SERVED_B=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['data'][0]['id'])" "${OUT}/models-b.json") fi + fi litellm() { # litellm x served port llm_port api_path printf 'model_list:\n - model_name: %s\n litellm_params:\n model: openai/%s\n api_base: http://127.0.0.1:%s%s\n api_key: not-needed\n timeout: 600\n' \ "${ALIAS}" "$2" "$3" "$5" > "${OUT}/litellm-$1.yaml" diff --git a/bench/eatflow/pairs.py b/bench/eatflow/pairs.py index 723f8452..d58c648a 100644 --- a/bench/eatflow/pairs.py +++ b/bench/eatflow/pairs.py @@ -17,12 +17,16 @@ pairs' pre-windows. bg_cpu, package W and throttle during the attempt are reported, never gated. -Load rule L3 (H50, H50o, H53; EXPERIMENTS.md "Crescent amendment A3"): the same +Load rule L3 (H50, H50o, H53, H62; EXPERIMENTS.md "Crescent amendment A3"): the same pre-windows, no match filter. Every complete pair enters the statistic; a balance gate (a sign-flip test of arm on each pre covariate, p < 0.01 gates) replaces it; a two-regressor adjustment and L2's matched subset are reported, never gated. The arm order of pair j is drawn from the registered seed (order), and each pairs directory carries one registration.json (register). Python 3 standard library only. + +A pod cgroup is a local cgroup v2 directory, or "exec:" for a pod on another +machine (H62, EXPERIMENTS.md "H62 pre-registration": each server runs in its own KVM guest): +the command prints that pod's cpu.stat, and the idle-arm check reads it like a local one. """ import glob import json @@ -103,9 +107,30 @@ def total_jiffies(stat_text, cpus): return tot +EXEC = "exec:" # a pod cgroup read through a command (H62: a pod in a KVM guest) +EXEC_TIMEOUT_S = 15.0 + + +def _exec_text(cmdline): + """stdout of a command line (shlex-split, no shell), or "" when it fails or times out.""" + argv = shlex.split(cmdline) + if not argv: + return "" + try: + p = subprocess.run(argv, capture_output=True, text=True, timeout=EXEC_TIMEOUT_S) + except (OSError, subprocess.SubprocessError): + return "" + return p.stdout if p.returncode == 0 else "" + + def cgroup_usage_usec(path): - """cpu.stat usage_usec of a cgroup v2 directory (None when unreadable).""" - for line in _read(os.path.join(path, "cpu.stat")).splitlines(): + """cpu.stat usage_usec of a cgroup v2 directory, or of "exec:", a command that + prints a pod's cpu.stat on another machine (None when unreadable).""" + if path.startswith(EXEC): + text = _exec_text(path[len(EXEC):]) + else: + text = _read(os.path.join(path, "cpu.stat")) + for line in text.splitlines(): k, _, v = line.partition(" ") if k == "usage_usec" and v.strip().isdigit(): return int(v) @@ -512,13 +537,14 @@ def boot_share(d, pred, boot=stats.BOOT, seed=stats.SEED, stat=None): # Rule L3 rows: the seed of their arm order, fixed in the pre-registration (amendment A3). ORDER_SEEDS = {"H50": "H50-crescent-2026-10-20", "H50o": "H50o-crescent-2026-10-20", - "H53": "H53-crescent-2026-10-20"} + "H53": "H53-crescent-2026-10-20", "H62": "H62-crescent-2026-10-29"} # Hypothesis id -> (design, load rule). H47b and H49b are H47 and H49 under rule L2; H50 is # H49b's design on the Xeon 6741P, H50o the same with arm B3 (EXPERIMENTS.md "H50-H52 pre-registration"); -# H53 the H49 statistic over solo-hold block pairs on the Arc Pro B70s (solo.py; block-level verdict, "H53 pre-registration"). +# H53 the H49 statistic over solo-hold block pairs on the Arc Pro B70s (solo.py; block-level verdict, "H53 pre-registration"); +# H62 H50's design with arm B5, Intel's EAS default, each server in a KVM guest ("H62 pre-registration"). DESIGNS = {"H47": ("H47", "L1"), "H49": ("H49", "L1"), "H47b": ("H47", "L2"), "H49b": ("H49", "L2"), - "H50": ("H49", "L3"), "H50o": ("H49", "L3"), "H53": ("H49", "L3")} + "H50": ("H49", "L3"), "H50o": ("H49", "L3"), "H53": ("H49", "L3"), "H62": ("H49", "L3")} # The ledger a rule-L3 row must be pre-registered in before any of its holds runs. @@ -814,13 +840,20 @@ def analyze(attempts, pairs, hyp, a, b, holds=None, registration=None, solo=Fals if __name__ == "__main__": # `pairs.py design HYP`: " " (live.sh reads the rule from here, DESIGNS only). # `pairs.py arm HYP N`: the arm solo hold N must run (solo-hold.sh checks it before the lock). + # `pairs.py usage SPEC...`: each pod cgroup's usage_usec, read once; exit 1 if one is unreadable + # (live.sh checks the H62 arms' exec: specs with it before a hold). try: if len(sys.argv) == 3 and sys.argv[1] == "design": print(*design(sys.argv[2])) elif len(sys.argv) == 4 and sys.argv[1] == "arm": design(sys.argv[2]) print(expected_arm(sys.argv[2], int(sys.argv[3]))) + elif len(sys.argv) >= 3 and sys.argv[1] == "usage": + got = [(spec, cgroup_usage_usec(spec)) for spec in sys.argv[2:]] + for spec, v in got: + print(f"usage_usec {v} {spec}") + sys.exit(1 if any(v is None for _, v in got) else 0) else: - raise SystemExit("usage: pairs.py design HYP | pairs.py arm HYP N") + raise SystemExit("usage: pairs.py design HYP | pairs.py arm HYP N | pairs.py usage SPEC...") except (ValueError, KeyError) as e: raise SystemExit(f"pairs.py: {e!r}") diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/fetch.log b/bench/eatflow/results/2026-10-20/H53/prehold/fetch.log new file mode 100644 index 00000000..b50e8c86 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/fetch.log @@ -0,0 +1,69 @@ +lock-taken 2026-10-29T21:44:37Z +6d5cab69b902d2d4b83d95af218c1ea069d0d9fc +Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. +Error while downloading from https://huggingface.co/Qwen/Qwen3.8-27B/resolve/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/model-00002-of-00018.safetensors: peer closed connection without sending complete message body (received 1344965215 bytes, expected 3043080328) +Trying to resume download... +✓ Downloaded + path: /home/devcloud/k3dgpu-bench/models/Qwen3.8-27B +Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. +✓ Downloaded + path: /home/devcloud/k3dgpu-bench/models/Qwen3.8-27B-int4-ov +ok size 1570 .gitattributes +ok size 11544 LICENSE +ok size 65012 README.md +ok size 8952 chat_template.jinja +ok size 4312 config.json +ok size 238 crc32.txt +ok size 202 generation_config.json +ok size 3353259 merges.txt +ok ba0ce20aae489ad196733da5064bcdf159a1fe84f53336648196e1ebb7751b1c model-00001-of-00018.safetensors +ok 06a148c01bfbe3faa14a5f184a7ff29a706f7ae1c8b2705d2058e26d17a001fb model-00002-of-00018.safetensors +ok 2e1bf62cbcd406eaa64b60d10353e1f0ef4039d0976e56f05cabe953454f9968 model-00003-of-00018.safetensors +ok 511e34063187882659753c4d93f3859f93c019fd438d8813071921c81d9a3f1a model-00004-of-00018.safetensors +ok 635cb53446dc74f219740fc59e18b774f877b803b9722e289ca62575a6efa701 model-00005-of-00018.safetensors +ok 0bc5214fac607f0e6cc92eec3789d4b8559410ef9fce66621ba8158e8410dae0 model-00006-of-00018.safetensors +ok 80b0c49033e9a0d5762562aa12f4acdb7f54da586f3d0110f28c48d91cf07892 model-00007-of-00018.safetensors +ok 7192c5b66185d3592927daabee1cc19e6f6e0ce75988ee20e824b624765fda79 model-00008-of-00018.safetensors +ok af3c48cc37af44f3db6ae0579baf019180d48d9c527caa0a1f03ff85813a56d8 model-00009-of-00018.safetensors +ok 163490a76f3bea3a40855b7efc04ce6d27afaf1a34f0bbde495b9491f76457c9 model-00010-of-00018.safetensors +ok 5f3ae1b948aeee39da77aec558e8236cd65fe4d7cb7686a76bb007acc563c6d8 model-00011-of-00018.safetensors +ok a3de1c7114677a8f5ac5c4892c90e8238ea5c1e2038c80e757dfc87c3902ca55 model-00012-of-00018.safetensors +ok 06ab79a41f74c9c5cb734816feb0c7fc364104b227165ee7391231e1155aa02a model-00013-of-00018.safetensors +ok 4138ed94603065ba884bbcadedb04d7718bb40117e85e6f5c6fc5b9c05b7a85b model-00014-of-00018.safetensors +ok 69224e27b9de4e7dbf6fc936c6eaae08447bda3b80a6c31a871ab451173afd22 model-00015-of-00018.safetensors +ok 73cb9a1089fb6155cb648609478d6633be8a5c7d9ca5a05bc8925ce8a553cefe model-00016-of-00018.safetensors +ok beb51f01056142ac4984bd800507b0dd0fd18de57f8e9ef6ea41d1a3598983a8 model-00017-of-00018.safetensors +ok 1d3479509e21494658f9b64d317f5ea8e55c4025d28c702d6c4d0b356ce8ea06 model-00018-of-00018.safetensors +ok size 112216 model.safetensors.index.json +ok size 390 preprocessor_config.json +ok 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3 tokenizer.json +ok size 17928 tokenizer_config.json +ok size 385 video_preprocessor_config.json +ok size 6722759 vocab.json +ok size 1570 .gitattributes +ok size 8321 README.md +ok size 8952 chat_template.jinja +ok size 3679 config.json +ok size 213 generation_config.json +ok size 1790 openvino_config.json +ok 48fc2c6a7a1056994982911eb9695826acef933c3d742364c56d16317a0b2a79 openvino_detokenizer.bin +ok size 16771 openvino_detokenizer.xml +ok 95aac72ac7286707c5e0fd9b120ea7f22c93f3ae17e7f1263442587d1d887d50 openvino_language_model.bin +ok size 9736672 openvino_language_model.xml +ok bc8ef47f252ad5a72be11e48ba6a23da50a67d1cef03396a7d902ca9aa1d5bc8 openvino_mtp_model.bin +ok size 210992 openvino_mtp_model.xml +ok e2099d0d83dcdfb00c378b51b50a52d28116aee71eb4f81cef0fe19ae37a085f openvino_text_embeddings_model.bin +ok size 5920 openvino_text_embeddings_model.xml +ok ca1ff37f580b558f0119b57824240adc92624d6d2ef6dd96ca2a24f4f1368c05 openvino_tokenizer.bin +ok size 35358 openvino_tokenizer.xml +ok 3792b6946c1f475aaac5d6f3960c7b912f612104e4f39ad68643b707370ed8e7 openvino_vision_embeddings_merger_model.bin +ok size 1330718 openvino_vision_embeddings_merger_model.xml +ok 5a8661b052abd07563d0f7148934d1fb77d5a842647151304287beb5dcee5541 openvino_vision_embeddings_model.bin +ok size 8610 openvino_vision_embeddings_model.xml +ok 82ccc97be0351cd82ebe1717fd7fa9adf72a44982a7ced8e8a845d869325d9e5 openvino_vision_embeddings_pos_model.bin +ok size 5887 openvino_vision_embeddings_pos_model.xml +ok size 390 preprocessor_config.json +ok size 1300 processor_config.json +ok 87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4 tokenizer.json +ok size 1098 tokenizer_config.json +rc=0 2026-10-29T21:55:52Z diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-post.txt b/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-post.txt new file mode 100644 index 00000000..5e145d28 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-post.txt @@ -0,0 +1,2 @@ +2026-10-30T00:37:52Z :30000 pass=1 before=2 after=3 content=OK +2026-10-30T00:38:03Z :30001 pass=1 before=3 after=4 content=OK diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-pre.txt b/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-pre.txt new file mode 100644 index 00000000..16e515a9 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-pre.txt @@ -0,0 +1,2 @@ +2026-10-30T00:14:45Z :30000 pass=1 before=161 after=162 content=OK +2026-10-30T00:14:45Z :30001 pass=1 before=57 after=58 content=OK diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.after.key.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.after.key.json new file mode 100644 index 00000000..63bd34fe --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.after.key.json @@ -0,0 +1 @@ +{"args": ["/entry.sh"], "binds": ["/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro", "/dev/dri:/dev/dri", "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", "/home/devcloud/hicache-shared:/hicache", "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", "/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro"], "devices": [{"CgroupPermissions": "rwm", "PathInContainer": "/dev/dri", "PathOnHost": "/dev/dri"}], "env": ["CCL_CONFIGURATION=cpu_gpu_dpcpp", "CCL_CONFIGURATION_PATH=", "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", "CTX=8192", "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", "DEBIAN_FRONTEND=noninteractive", "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", "DPLROOT=/opt/intel/oneapi/dpl/2022.13", "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", "KV_PORT=5557", "LANG=C.UTF-8", "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", "MKLROOT=/opt/intel/oneapi/mkl/2026.1", "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", "ONEAPI_ROOT=/opt/intel/oneapi", "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", "PORT=30000", "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", "SETVARS_COMPLETED=1", "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", "SGLANG_CHUNKED_PREFILL=4096", "SGLANG_CONTEXT_LEN=262144", "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", "SGLANG_HTTP_PORT=30000", "SGLANG_MAX_RUNNING=64", "SGLANG_MEM_FRACTION=0.85", "SGLANG_MODEL=/model", "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", "SGLANG_PAGE_SIZE=64", "SGLANG_TP_SIZE=2", "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", "TP_SIZE=2", "UMF_ROOT=/opt/intel/oneapi/umf/1.1", "UV_PYTHON_INSTALL_DIR=/opt/uv/python", "VIRTUAL_ENV=/opt/venv", "XPU_DENSE_INT8=1", "XPU_DENSE_INT8_MAX_M=8", "XPU_GDN_DECODE_BV=16", "XPU_KERNELS=topk,gemv", "XPU_MAMBA_PAGED=1", "XPU_MOE_ALIGN_SKIP=1", "XPU_MOE_BUILD=/opt/xpu/build-gemv", "XPU_REPO=/opt/xpu", "XPU_SYMM_AR=1", "XPU_SYMM_AR_MAX_BYTES=131072", "XPU_TOPK_BUILD=/opt/xpu/build", "XPU_UNROLL_SPEC_VERIFY=1", "ZE_AFFINITY_MASK=0,1"], "image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", "network": "bridge", "path": "bash", "ports": {"30000/tcp": [{"HostIp": "", "HostPort": "30000"}], "5557/tcp": [{"HostIp": "", "HostPort": "5557"}]}, "restart": {"MaximumRetryCount": 0, "Name": "no"}} diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.json new file mode 100644 index 00000000..21e2ee78 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.json @@ -0,0 +1,491 @@ +[ + { + "Id": "e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127", + "Created": "2026-10-19T07:43:07.036551199Z", + "Path": "bash", + "Args": [ + "/entry.sh" + ], + "State": { + "Status": "running", + "Running": true, + "Paused": false, + "Restarting": false, + "OOMKilled": false, + "Dead": false, + "Pid": 3898329, + "ExitCode": 0, + "Error": "", + "StartedAt": "2026-10-19T07:43:07.057267648Z", + "FinishedAt": "0001-01-01T00:00:00Z" + }, + "Image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", + "ResolvConfPath": "/var/lib/docker/containers/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127/resolv.conf", + "HostnamePath": "/var/lib/docker/containers/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127/hostname", + "HostsPath": "/var/lib/docker/containers/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127/hosts", + "LogPath": "/var/lib/docker/containers/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127-json.log", + "Name": "/qwen38-a-hr", + "RestartCount": 0, + "Driver": "overlayfs", + "Platform": "linux", + "MountLabel": "", + "ProcessLabel": "", + "AppArmorProfile": "docker-default", + "ExecIDs": null, + "HostConfig": { + "Binds": [ + "/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", + "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro", + "/dev/dri:/dev/dri", + "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", + "/home/devcloud/hicache-shared:/hicache", + "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", + "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", + "/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", + "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro" + ], + "ContainerIDFile": "", + "LogConfig": { + "Type": "json-file", + "Config": {} + }, + "NetworkMode": "bridge", + "PortBindings": { + "30000/tcp": [ + { + "HostIp": "", + "HostPort": "30000" + } + ], + "5557/tcp": [ + { + "HostIp": "", + "HostPort": "5557" + } + ] + }, + "RestartPolicy": { + "Name": "no", + "MaximumRetryCount": 0 + }, + "AutoRemove": false, + "VolumeDriver": "", + "VolumesFrom": null, + "ConsoleSize": [ + 0, + 0 + ], + "CapAdd": null, + "CapDrop": null, + "CgroupnsMode": "private", + "Dns": null, + "DnsOptions": [], + "DnsSearch": [], + "ExtraHosts": null, + "GroupAdd": null, + "IpcMode": "host", + "Cgroup": "", + "Links": null, + "OomScoreAdj": 0, + "PidMode": "", + "Privileged": false, + "PublishAllPorts": false, + "ReadonlyRootfs": false, + "SecurityOpt": [ + "label=disable" + ], + "UTSMode": "", + "UsernsMode": "", + "ShmSize": 34359738368, + "Runtime": "runc", + "Isolation": "", + "CpuShares": 0, + "Memory": 0, + "NanoCpus": 0, + "CgroupParent": "", + "BlkioWeight": 0, + "BlkioWeightDevice": [], + "BlkioDeviceReadBps": [], + "BlkioDeviceWriteBps": [], + "BlkioDeviceReadIOps": [], + "BlkioDeviceWriteIOps": [], + "CpuPeriod": 0, + "CpuQuota": 0, + "CpuRealtimePeriod": 0, + "CpuRealtimeRuntime": 0, + "CpusetCpus": "", + "CpusetMems": "", + "Devices": [ + { + "PathOnHost": "/dev/dri", + "PathInContainer": "/dev/dri", + "CgroupPermissions": "rwm" + } + ], + "DeviceCgroupRules": null, + "DeviceRequests": null, + "MemoryReservation": 0, + "MemorySwap": 0, + "MemorySwappiness": null, + "OomKillDisable": null, + "PidsLimit": null, + "Ulimits": null, + "CpuCount": 0, + "CpuPercent": 0, + "IOMaximumIOps": 0, + "IOMaximumBandwidth": 0, + "MaskedPaths": [ + "/proc/acpi", + "/proc/asound", + "/proc/interrupts", + "/proc/kcore", + "/proc/keys", + "/proc/latency_stats", + "/proc/sched_debug", + "/proc/scsi", + "/proc/timer_list", + "/proc/timer_stats", + "/sys/devices/virtual/powercap", + "/sys/firmware", + "/sys/devices/system/cpu/cpu0/thermal_throttle", + "/sys/devices/system/cpu/cpu1/thermal_throttle", + "/sys/devices/system/cpu/cpu2/thermal_throttle", + "/sys/devices/system/cpu/cpu3/thermal_throttle", + "/sys/devices/system/cpu/cpu4/thermal_throttle", + "/sys/devices/system/cpu/cpu5/thermal_throttle", + "/sys/devices/system/cpu/cpu6/thermal_throttle", + "/sys/devices/system/cpu/cpu7/thermal_throttle", + "/sys/devices/system/cpu/cpu8/thermal_throttle", + "/sys/devices/system/cpu/cpu9/thermal_throttle", + "/sys/devices/system/cpu/cpu10/thermal_throttle", + "/sys/devices/system/cpu/cpu11/thermal_throttle", + "/sys/devices/system/cpu/cpu12/thermal_throttle", + "/sys/devices/system/cpu/cpu13/thermal_throttle", + "/sys/devices/system/cpu/cpu14/thermal_throttle", + "/sys/devices/system/cpu/cpu15/thermal_throttle", + "/sys/devices/system/cpu/cpu16/thermal_throttle", + "/sys/devices/system/cpu/cpu17/thermal_throttle", + "/sys/devices/system/cpu/cpu18/thermal_throttle", + "/sys/devices/system/cpu/cpu19/thermal_throttle", + "/sys/devices/system/cpu/cpu20/thermal_throttle", + "/sys/devices/system/cpu/cpu21/thermal_throttle", + "/sys/devices/system/cpu/cpu22/thermal_throttle", + "/sys/devices/system/cpu/cpu23/thermal_throttle", + "/sys/devices/system/cpu/cpu24/thermal_throttle", + "/sys/devices/system/cpu/cpu25/thermal_throttle", + "/sys/devices/system/cpu/cpu26/thermal_throttle", + "/sys/devices/system/cpu/cpu27/thermal_throttle", + "/sys/devices/system/cpu/cpu28/thermal_throttle", + "/sys/devices/system/cpu/cpu29/thermal_throttle", + "/sys/devices/system/cpu/cpu30/thermal_throttle", + "/sys/devices/system/cpu/cpu31/thermal_throttle", + "/sys/devices/system/cpu/cpu32/thermal_throttle", + "/sys/devices/system/cpu/cpu33/thermal_throttle", + "/sys/devices/system/cpu/cpu34/thermal_throttle", + "/sys/devices/system/cpu/cpu35/thermal_throttle", + "/sys/devices/system/cpu/cpu36/thermal_throttle", + "/sys/devices/system/cpu/cpu37/thermal_throttle", + "/sys/devices/system/cpu/cpu38/thermal_throttle", + "/sys/devices/system/cpu/cpu39/thermal_throttle", + "/sys/devices/system/cpu/cpu40/thermal_throttle", + "/sys/devices/system/cpu/cpu41/thermal_throttle", + "/sys/devices/system/cpu/cpu42/thermal_throttle", + "/sys/devices/system/cpu/cpu43/thermal_throttle", + "/sys/devices/system/cpu/cpu44/thermal_throttle", + "/sys/devices/system/cpu/cpu45/thermal_throttle", + "/sys/devices/system/cpu/cpu46/thermal_throttle", + "/sys/devices/system/cpu/cpu47/thermal_throttle", + "/sys/devices/system/cpu/cpu48/thermal_throttle", + "/sys/devices/system/cpu/cpu49/thermal_throttle", + "/sys/devices/system/cpu/cpu50/thermal_throttle", + "/sys/devices/system/cpu/cpu51/thermal_throttle", + "/sys/devices/system/cpu/cpu52/thermal_throttle", + "/sys/devices/system/cpu/cpu53/thermal_throttle", + "/sys/devices/system/cpu/cpu54/thermal_throttle", + "/sys/devices/system/cpu/cpu55/thermal_throttle", + "/sys/devices/system/cpu/cpu56/thermal_throttle", + "/sys/devices/system/cpu/cpu57/thermal_throttle", + "/sys/devices/system/cpu/cpu58/thermal_throttle", + "/sys/devices/system/cpu/cpu59/thermal_throttle", + "/sys/devices/system/cpu/cpu60/thermal_throttle", + "/sys/devices/system/cpu/cpu61/thermal_throttle", + "/sys/devices/system/cpu/cpu62/thermal_throttle", + "/sys/devices/system/cpu/cpu63/thermal_throttle", + "/sys/devices/system/cpu/cpu64/thermal_throttle", + "/sys/devices/system/cpu/cpu65/thermal_throttle", + "/sys/devices/system/cpu/cpu66/thermal_throttle", + "/sys/devices/system/cpu/cpu67/thermal_throttle", + "/sys/devices/system/cpu/cpu68/thermal_throttle", + "/sys/devices/system/cpu/cpu69/thermal_throttle", + "/sys/devices/system/cpu/cpu70/thermal_throttle", + "/sys/devices/system/cpu/cpu71/thermal_throttle", + "/sys/devices/system/cpu/cpu72/thermal_throttle", + "/sys/devices/system/cpu/cpu73/thermal_throttle", + "/sys/devices/system/cpu/cpu74/thermal_throttle", + "/sys/devices/system/cpu/cpu75/thermal_throttle", + "/sys/devices/system/cpu/cpu76/thermal_throttle", + "/sys/devices/system/cpu/cpu77/thermal_throttle", + "/sys/devices/system/cpu/cpu78/thermal_throttle", + "/sys/devices/system/cpu/cpu79/thermal_throttle", + "/sys/devices/system/cpu/cpu80/thermal_throttle", + "/sys/devices/system/cpu/cpu81/thermal_throttle", + "/sys/devices/system/cpu/cpu82/thermal_throttle", + "/sys/devices/system/cpu/cpu83/thermal_throttle", + "/sys/devices/system/cpu/cpu84/thermal_throttle", + "/sys/devices/system/cpu/cpu85/thermal_throttle", + "/sys/devices/system/cpu/cpu86/thermal_throttle", + "/sys/devices/system/cpu/cpu87/thermal_throttle", + "/sys/devices/system/cpu/cpu88/thermal_throttle", + "/sys/devices/system/cpu/cpu89/thermal_throttle", + "/sys/devices/system/cpu/cpu90/thermal_throttle", + "/sys/devices/system/cpu/cpu91/thermal_throttle", + "/sys/devices/system/cpu/cpu92/thermal_throttle", + "/sys/devices/system/cpu/cpu93/thermal_throttle", + "/sys/devices/system/cpu/cpu94/thermal_throttle", + "/sys/devices/system/cpu/cpu95/thermal_throttle" + ], + "ReadonlyPaths": [ + "/proc/bus", + "/proc/fs", + "/proc/irq", + "/proc/sys", + "/proc/sysrq-trigger" + ] + }, + "Storage": { + "RootFS": { + "Snapshot": { + "Name": "overlayfs" + } + } + }, + "Mounts": [ + { + "Type": "bind", + "Source": "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh", + "Destination": "/entry.sh", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/hicache-shared", + "Destination": "/hicache", + "Mode": "", + "RW": true, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4", + "Destination": "/model", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/dev/dri", + "Destination": "/dev/dri", + "Mode": "", + "RW": true, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/empty-memcache-pyc", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-mph.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/empty-pycache", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-common.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-mha.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + } + ], + "Config": { + "Hostname": "e4e0a92af9b8", + "Domainname": "", + "User": "root", + "AttachStdin": false, + "AttachStdout": false, + "AttachStderr": false, + "ExposedPorts": { + "30000/tcp": {}, + "5557/tcp": {} + }, + "Tty": false, + "OpenStdin": false, + "StdinOnce": false, + "Env": [ + "XPU_SYMM_AR=1", + "KV_PORT=5557", + "TP_SIZE=2", + "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", + "XPU_MAMBA_PAGED=1", + "XPU_DENSE_INT8_MAX_M=8", + "XPU_SYMM_AR_MAX_BYTES=131072", + "XPU_UNROLL_SPEC_VERIFY=1", + "PORT=30000", + "CTX=8192", + "XPU_DENSE_INT8=1", + "ZE_AFFINITY_MASK=0,1", + "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", + "XPU_GDN_DECODE_BV=16", + "XPU_MOE_ALIGN_SKIP=1", + "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "LANG=C.UTF-8", + "CCL_CONFIGURATION=cpu_gpu_dpcpp", + "CCL_CONFIGURATION_PATH=", + "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", + "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", + "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", + "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", + "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", + "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", + "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", + "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", + "DPLROOT=/opt/intel/oneapi/dpl/2022.13", + "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", + "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", + "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", + "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", + "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", + "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", + "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", + "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", + "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", + "MKLROOT=/opt/intel/oneapi/mkl/2026.1", + "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", + "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", + "ONEAPI_ROOT=/opt/intel/oneapi", + "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", + "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", + "SETVARS_COMPLETED=1", + "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", + "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", + "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", + "UMF_ROOT=/opt/intel/oneapi/umf/1.1", + "DEBIAN_FRONTEND=noninteractive", + "VIRTUAL_ENV=/opt/venv", + "UV_PYTHON_INSTALL_DIR=/opt/uv/python", + "XPU_TOPK_BUILD=/opt/xpu/build", + "XPU_MOE_BUILD=/opt/xpu/build-gemv", + "XPU_REPO=/opt/xpu", + "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", + "XPU_KERNELS=topk,gemv", + "SGLANG_TP_SIZE=2", + "SGLANG_HTTP_PORT=30000", + "SGLANG_CONTEXT_LEN=262144", + "SGLANG_MEM_FRACTION=0.85", + "SGLANG_CHUNKED_PREFILL=4096", + "SGLANG_MAX_RUNNING=64", + "SGLANG_PAGE_SIZE=64", + "SGLANG_MODEL=/model", + "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1" + ], + "Cmd": [ + "/entry.sh" + ], + "Image": "rahulunair/sglang-xpu@sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", + "Volumes": null, + "WorkingDir": "/sgl-workspace/sglang", + "Entrypoint": [ + "bash" + ], + "Labels": { + "org.opencontainers.image.description": "SGLang serving Qwen3.8-27B AWQ-INT4 or FP8 on four Intel Arc Pro B70 GPUs, with MTP and DFlash2 speculative decoding through the checkpoint's native 262,144-token context.", + "org.opencontainers.image.title": "Qwen3.8-27B on Intel Arc Pro", + "org.opencontainers.image.version": "24.04", + "xpu.qwen38.base": "rahulunair/sglang-xpu:qwen3.8-27b-20260819", + "xpu.qwen38.built": "20260820", + "xpu.qwen38.dflash2_source": "e5a3e4d30fa7abda95bafd2d697f9f9c48566114" + } + }, + "NetworkSettings": { + "SandboxID": "0a87eb278cbd20310832479bc36cefc9c3ff495719de11a71f74bf1a2d2b212d", + "SandboxKey": "/var/run/docker/netns/0a87eb278cbd", + "Ports": { + "30000/tcp": [ + { + "HostIp": "0.0.0.0", + "HostPort": "30000" + }, + { + "HostIp": "::", + "HostPort": "30000" + } + ], + "5557/tcp": [ + { + "HostIp": "0.0.0.0", + "HostPort": "5557" + }, + { + "HostIp": "::", + "HostPort": "5557" + } + ] + }, + "Networks": { + "bridge": { + "IPAMConfig": null, + "Links": null, + "Aliases": null, + "DriverOpts": null, + "GwPriority": 0, + "NetworkID": "079840671febd9bc1caa55e83e63558724adbe92299b1abb5777fe9fa72b5975", + "EndpointID": "2633a049bbcd307089aa4f67b15ce5962783ed209543218d550d6c1e760462fd", + "Gateway": "172.17.0.1", + "IPAddress": "172.17.0.2", + "MacAddress": "6a:23:3b:de:2a:39", + "IPPrefixLen": 16, + "IPv6Gateway": "", + "GlobalIPv6Address": "", + "GlobalIPv6PrefixLen": 0, + "DNSNames": null + } + } + }, + "ImageManifestDescriptor": { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "digest": "sha256:70aab3fb164f7941db3ec1b45be934bad70ff720f4013fd5b4429b3d90668e5a", + "size": 5653, + "platform": { + "architecture": "amd64", + "os": "linux" + } + } + } +] diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.key.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.key.json new file mode 100644 index 00000000..63bd34fe --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.key.json @@ -0,0 +1 @@ +{"args": ["/entry.sh"], "binds": ["/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro", "/dev/dri:/dev/dri", "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", "/home/devcloud/hicache-shared:/hicache", "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", "/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro"], "devices": [{"CgroupPermissions": "rwm", "PathInContainer": "/dev/dri", "PathOnHost": "/dev/dri"}], "env": ["CCL_CONFIGURATION=cpu_gpu_dpcpp", "CCL_CONFIGURATION_PATH=", "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", "CTX=8192", "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", "DEBIAN_FRONTEND=noninteractive", "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", "DPLROOT=/opt/intel/oneapi/dpl/2022.13", "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", "KV_PORT=5557", "LANG=C.UTF-8", "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", "MKLROOT=/opt/intel/oneapi/mkl/2026.1", "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", "ONEAPI_ROOT=/opt/intel/oneapi", "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", "PORT=30000", "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", "SETVARS_COMPLETED=1", "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", "SGLANG_CHUNKED_PREFILL=4096", "SGLANG_CONTEXT_LEN=262144", "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", "SGLANG_HTTP_PORT=30000", "SGLANG_MAX_RUNNING=64", "SGLANG_MEM_FRACTION=0.85", "SGLANG_MODEL=/model", "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", "SGLANG_PAGE_SIZE=64", "SGLANG_TP_SIZE=2", "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", "TP_SIZE=2", "UMF_ROOT=/opt/intel/oneapi/umf/1.1", "UV_PYTHON_INSTALL_DIR=/opt/uv/python", "VIRTUAL_ENV=/opt/venv", "XPU_DENSE_INT8=1", "XPU_DENSE_INT8_MAX_M=8", "XPU_GDN_DECODE_BV=16", "XPU_KERNELS=topk,gemv", "XPU_MAMBA_PAGED=1", "XPU_MOE_ALIGN_SKIP=1", "XPU_MOE_BUILD=/opt/xpu/build-gemv", "XPU_REPO=/opt/xpu", "XPU_SYMM_AR=1", "XPU_SYMM_AR_MAX_BYTES=131072", "XPU_TOPK_BUILD=/opt/xpu/build", "XPU_UNROLL_SPEC_VERIFY=1", "ZE_AFFINITY_MASK=0,1"], "image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", "network": "bridge", "path": "bash", "ports": {"30000/tcp": [{"HostIp": "", "HostPort": "30000"}], "5557/tcp": [{"HostIp": "", "HostPort": "5557"}]}, "restart": {"MaximumRetryCount": 0, "Name": "no"}} diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.after.key.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.after.key.json new file mode 100644 index 00000000..c8dd0d9d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.after.key.json @@ -0,0 +1 @@ +{"args": ["/entry.sh"], "binds": ["/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", "/dev/dri:/dev/dri", "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", "/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro", "/home/devcloud/hicache-shared:/hicache", "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro"], "devices": [{"CgroupPermissions": "rwm", "PathInContainer": "/dev/dri", "PathOnHost": "/dev/dri"}], "env": ["CCL_CONFIGURATION=cpu_gpu_dpcpp", "CCL_CONFIGURATION_PATH=", "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", "CTX=8192", "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", "DEBIAN_FRONTEND=noninteractive", "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", "DPLROOT=/opt/intel/oneapi/dpl/2022.13", "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", "KV_PORT=5558", "LANG=C.UTF-8", "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", "MKLROOT=/opt/intel/oneapi/mkl/2026.1", "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", "ONEAPI_ROOT=/opt/intel/oneapi", "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", "PORT=30001", "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", "SETVARS_COMPLETED=1", "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", "SGLANG_CHUNKED_PREFILL=4096", "SGLANG_CONTEXT_LEN=262144", "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", "SGLANG_HTTP_PORT=30000", "SGLANG_MAX_RUNNING=64", "SGLANG_MEM_FRACTION=0.85", "SGLANG_MODEL=/model", "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", "SGLANG_PAGE_SIZE=64", "SGLANG_TP_SIZE=2", "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", "TP_SIZE=2", "UMF_ROOT=/opt/intel/oneapi/umf/1.1", "UV_PYTHON_INSTALL_DIR=/opt/uv/python", "VIRTUAL_ENV=/opt/venv", "XPU_DENSE_INT8=1", "XPU_DENSE_INT8_MAX_M=8", "XPU_GDN_DECODE_BV=16", "XPU_KERNELS=topk,gemv", "XPU_MAMBA_PAGED=1", "XPU_MOE_ALIGN_SKIP=1", "XPU_MOE_BUILD=/opt/xpu/build-gemv", "XPU_REPO=/opt/xpu", "XPU_SYMM_AR=1", "XPU_SYMM_AR_MAX_BYTES=131072", "XPU_TOPK_BUILD=/opt/xpu/build", "XPU_UNROLL_SPEC_VERIFY=1", "ZE_AFFINITY_MASK=2,3"], "image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", "network": "bridge", "path": "bash", "ports": {"30001/tcp": [{"HostIp": "", "HostPort": "30001"}], "5558/tcp": [{"HostIp": "", "HostPort": "5558"}]}, "restart": {"MaximumRetryCount": 0, "Name": "no"}} diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.json new file mode 100644 index 00000000..42aeb1bd --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.json @@ -0,0 +1,493 @@ +[ + { + "Id": "e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80", + "Created": "2026-10-19T07:43:07.195804765Z", + "Path": "bash", + "Args": [ + "/entry.sh" + ], + "State": { + "Status": "running", + "Running": true, + "Paused": false, + "Restarting": false, + "OOMKilled": false, + "Dead": false, + "Pid": 528967, + "ExitCode": 0, + "Error": "", + "StartedAt": "2026-10-21T21:57:40.048150052Z", + "FinishedAt": "2026-10-21T21:12:31.94913436Z" + }, + "Image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", + "ResolvConfPath": "/var/lib/docker/containers/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80/resolv.conf", + "HostnamePath": "/var/lib/docker/containers/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80/hostname", + "HostsPath": "/var/lib/docker/containers/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80/hosts", + "LogPath": "/var/lib/docker/containers/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80-json.log", + "Name": "/qwen38-b-hr", + "RestartCount": 0, + "Driver": "overlayfs", + "Platform": "linux", + "MountLabel": "", + "ProcessLabel": "", + "AppArmorProfile": "docker-default", + "ExecIDs": null, + "HostConfig": { + "Binds": [ + "/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", + "/dev/dri:/dev/dri", + "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", + "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", + "/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", + "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro", + "/home/devcloud/hicache-shared:/hicache", + "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", + "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro" + ], + "ContainerIDFile": "", + "LogConfig": { + "Type": "json-file", + "Config": {} + }, + "NetworkMode": "bridge", + "PortBindings": { + "30001/tcp": [ + { + "HostIp": "", + "HostPort": "30001" + } + ], + "5558/tcp": [ + { + "HostIp": "", + "HostPort": "5558" + } + ] + }, + "RestartPolicy": { + "Name": "no", + "MaximumRetryCount": 0 + }, + "AutoRemove": false, + "VolumeDriver": "", + "VolumesFrom": null, + "ConsoleSize": [ + 0, + 0 + ], + "CapAdd": null, + "CapDrop": null, + "CgroupnsMode": "private", + "Dns": null, + "DnsOptions": [], + "DnsSearch": [], + "ExtraHosts": null, + "GroupAdd": null, + "IpcMode": "host", + "Cgroup": "", + "Links": null, + "OomScoreAdj": 0, + "PidMode": "", + "Privileged": false, + "PublishAllPorts": false, + "ReadonlyRootfs": false, + "SecurityOpt": [ + "label=disable" + ], + "UTSMode": "", + "UsernsMode": "", + "ShmSize": 34359738368, + "Runtime": "runc", + "Isolation": "", + "CpuShares": 0, + "Memory": 0, + "NanoCpus": 0, + "CgroupParent": "", + "BlkioWeight": 0, + "BlkioWeightDevice": [], + "BlkioDeviceReadBps": [], + "BlkioDeviceWriteBps": [], + "BlkioDeviceReadIOps": [], + "BlkioDeviceWriteIOps": [], + "CpuPeriod": 0, + "CpuQuota": 0, + "CpuRealtimePeriod": 0, + "CpuRealtimeRuntime": 0, + "CpusetCpus": "", + "CpusetMems": "", + "Devices": [ + { + "PathOnHost": "/dev/dri", + "PathInContainer": "/dev/dri", + "CgroupPermissions": "rwm" + } + ], + "DeviceCgroupRules": null, + "DeviceRequests": null, + "MemoryReservation": 0, + "MemorySwap": 0, + "MemorySwappiness": null, + "OomKillDisable": null, + "PidsLimit": null, + "Ulimits": null, + "CpuCount": 0, + "CpuPercent": 0, + "IOMaximumIOps": 0, + "IOMaximumBandwidth": 0, + "MaskedPaths": [ + "/proc/acpi", + "/proc/asound", + "/proc/interrupts", + "/proc/kcore", + "/proc/keys", + "/proc/latency_stats", + "/proc/sched_debug", + "/proc/scsi", + "/proc/timer_list", + "/proc/timer_stats", + "/sys/devices/virtual/powercap", + "/sys/firmware", + "/sys/devices/system/cpu/cpu0/thermal_throttle", + "/sys/devices/system/cpu/cpu1/thermal_throttle", + "/sys/devices/system/cpu/cpu2/thermal_throttle", + "/sys/devices/system/cpu/cpu3/thermal_throttle", + "/sys/devices/system/cpu/cpu4/thermal_throttle", + "/sys/devices/system/cpu/cpu5/thermal_throttle", + "/sys/devices/system/cpu/cpu6/thermal_throttle", + "/sys/devices/system/cpu/cpu7/thermal_throttle", + "/sys/devices/system/cpu/cpu8/thermal_throttle", + "/sys/devices/system/cpu/cpu9/thermal_throttle", + "/sys/devices/system/cpu/cpu10/thermal_throttle", + "/sys/devices/system/cpu/cpu11/thermal_throttle", + "/sys/devices/system/cpu/cpu12/thermal_throttle", + "/sys/devices/system/cpu/cpu13/thermal_throttle", + "/sys/devices/system/cpu/cpu14/thermal_throttle", + "/sys/devices/system/cpu/cpu15/thermal_throttle", + "/sys/devices/system/cpu/cpu16/thermal_throttle", + "/sys/devices/system/cpu/cpu17/thermal_throttle", + "/sys/devices/system/cpu/cpu18/thermal_throttle", + "/sys/devices/system/cpu/cpu19/thermal_throttle", + "/sys/devices/system/cpu/cpu20/thermal_throttle", + "/sys/devices/system/cpu/cpu21/thermal_throttle", + "/sys/devices/system/cpu/cpu22/thermal_throttle", + "/sys/devices/system/cpu/cpu23/thermal_throttle", + "/sys/devices/system/cpu/cpu24/thermal_throttle", + "/sys/devices/system/cpu/cpu25/thermal_throttle", + "/sys/devices/system/cpu/cpu26/thermal_throttle", + "/sys/devices/system/cpu/cpu27/thermal_throttle", + "/sys/devices/system/cpu/cpu28/thermal_throttle", + "/sys/devices/system/cpu/cpu29/thermal_throttle", + "/sys/devices/system/cpu/cpu30/thermal_throttle", + "/sys/devices/system/cpu/cpu31/thermal_throttle", + "/sys/devices/system/cpu/cpu32/thermal_throttle", + "/sys/devices/system/cpu/cpu33/thermal_throttle", + "/sys/devices/system/cpu/cpu34/thermal_throttle", + "/sys/devices/system/cpu/cpu35/thermal_throttle", + "/sys/devices/system/cpu/cpu36/thermal_throttle", + "/sys/devices/system/cpu/cpu37/thermal_throttle", + "/sys/devices/system/cpu/cpu38/thermal_throttle", + "/sys/devices/system/cpu/cpu39/thermal_throttle", + "/sys/devices/system/cpu/cpu40/thermal_throttle", + "/sys/devices/system/cpu/cpu41/thermal_throttle", + "/sys/devices/system/cpu/cpu42/thermal_throttle", + "/sys/devices/system/cpu/cpu43/thermal_throttle", + "/sys/devices/system/cpu/cpu44/thermal_throttle", + "/sys/devices/system/cpu/cpu45/thermal_throttle", + "/sys/devices/system/cpu/cpu46/thermal_throttle", + "/sys/devices/system/cpu/cpu47/thermal_throttle", + "/sys/devices/system/cpu/cpu48/thermal_throttle", + "/sys/devices/system/cpu/cpu49/thermal_throttle", + "/sys/devices/system/cpu/cpu50/thermal_throttle", + "/sys/devices/system/cpu/cpu51/thermal_throttle", + "/sys/devices/system/cpu/cpu52/thermal_throttle", + "/sys/devices/system/cpu/cpu53/thermal_throttle", + "/sys/devices/system/cpu/cpu54/thermal_throttle", + "/sys/devices/system/cpu/cpu55/thermal_throttle", + "/sys/devices/system/cpu/cpu56/thermal_throttle", + "/sys/devices/system/cpu/cpu57/thermal_throttle", + "/sys/devices/system/cpu/cpu58/thermal_throttle", + "/sys/devices/system/cpu/cpu59/thermal_throttle", + "/sys/devices/system/cpu/cpu60/thermal_throttle", + "/sys/devices/system/cpu/cpu61/thermal_throttle", + "/sys/devices/system/cpu/cpu62/thermal_throttle", + "/sys/devices/system/cpu/cpu63/thermal_throttle", + "/sys/devices/system/cpu/cpu64/thermal_throttle", + "/sys/devices/system/cpu/cpu65/thermal_throttle", + "/sys/devices/system/cpu/cpu66/thermal_throttle", + "/sys/devices/system/cpu/cpu67/thermal_throttle", + "/sys/devices/system/cpu/cpu68/thermal_throttle", + "/sys/devices/system/cpu/cpu69/thermal_throttle", + "/sys/devices/system/cpu/cpu70/thermal_throttle", + "/sys/devices/system/cpu/cpu71/thermal_throttle", + "/sys/devices/system/cpu/cpu72/thermal_throttle", + "/sys/devices/system/cpu/cpu73/thermal_throttle", + "/sys/devices/system/cpu/cpu74/thermal_throttle", + "/sys/devices/system/cpu/cpu75/thermal_throttle", + "/sys/devices/system/cpu/cpu76/thermal_throttle", + "/sys/devices/system/cpu/cpu77/thermal_throttle", + "/sys/devices/system/cpu/cpu78/thermal_throttle", + "/sys/devices/system/cpu/cpu79/thermal_throttle", + "/sys/devices/system/cpu/cpu80/thermal_throttle", + "/sys/devices/system/cpu/cpu81/thermal_throttle", + "/sys/devices/system/cpu/cpu82/thermal_throttle", + "/sys/devices/system/cpu/cpu83/thermal_throttle", + "/sys/devices/system/cpu/cpu84/thermal_throttle", + "/sys/devices/system/cpu/cpu85/thermal_throttle", + "/sys/devices/system/cpu/cpu86/thermal_throttle", + "/sys/devices/system/cpu/cpu87/thermal_throttle", + "/sys/devices/system/cpu/cpu88/thermal_throttle", + "/sys/devices/system/cpu/cpu89/thermal_throttle", + "/sys/devices/system/cpu/cpu90/thermal_throttle", + "/sys/devices/system/cpu/cpu91/thermal_throttle", + "/sys/devices/system/cpu/cpu92/thermal_throttle", + "/sys/devices/system/cpu/cpu93/thermal_throttle", + "/sys/devices/system/cpu/cpu94/thermal_throttle", + "/sys/devices/system/cpu/cpu95/thermal_throttle" + ], + "ReadonlyPaths": [ + "/proc/bus", + "/proc/fs", + "/proc/irq", + "/proc/sys", + "/proc/sysrq-trigger" + ] + }, + "Storage": { + "RootFS": { + "Snapshot": { + "Name": "overlayfs" + } + } + }, + "Mounts": [ + { + "Type": "bind", + "Source": "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh", + "Destination": "/entry.sh", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/hicache-shared", + "Destination": "/hicache", + "Mode": "", + "RW": true, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4", + "Destination": "/model", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/dev/dri", + "Destination": "/dev/dri", + "Mode": "", + "RW": true, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/empty-memcache-pyc", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-mph.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/empty-pycache", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-common.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-mha.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + } + ], + "Config": { + "Hostname": "e4d6195699d9", + "Domainname": "", + "User": "root", + "AttachStdin": false, + "AttachStdout": false, + "AttachStderr": false, + "ExposedPorts": { + "30000/tcp": {}, + "30001/tcp": {}, + "5558/tcp": {} + }, + "Tty": false, + "OpenStdin": false, + "StdinOnce": false, + "Env": [ + "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", + "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", + "XPU_DENSE_INT8=1", + "XPU_SYMM_AR=1", + "XPU_MAMBA_PAGED=1", + "XPU_DENSE_INT8_MAX_M=8", + "CTX=8192", + "KV_PORT=5558", + "ZE_AFFINITY_MASK=2,3", + "XPU_SYMM_AR_MAX_BYTES=131072", + "XPU_UNROLL_SPEC_VERIFY=1", + "XPU_GDN_DECODE_BV=16", + "XPU_MOE_ALIGN_SKIP=1", + "PORT=30001", + "TP_SIZE=2", + "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "LANG=C.UTF-8", + "CCL_CONFIGURATION=cpu_gpu_dpcpp", + "CCL_CONFIGURATION_PATH=", + "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", + "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", + "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", + "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", + "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", + "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", + "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", + "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", + "DPLROOT=/opt/intel/oneapi/dpl/2022.13", + "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", + "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", + "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", + "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", + "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", + "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", + "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", + "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", + "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", + "MKLROOT=/opt/intel/oneapi/mkl/2026.1", + "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", + "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", + "ONEAPI_ROOT=/opt/intel/oneapi", + "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", + "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", + "SETVARS_COMPLETED=1", + "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", + "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", + "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", + "UMF_ROOT=/opt/intel/oneapi/umf/1.1", + "DEBIAN_FRONTEND=noninteractive", + "VIRTUAL_ENV=/opt/venv", + "UV_PYTHON_INSTALL_DIR=/opt/uv/python", + "XPU_TOPK_BUILD=/opt/xpu/build", + "XPU_MOE_BUILD=/opt/xpu/build-gemv", + "XPU_REPO=/opt/xpu", + "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", + "XPU_KERNELS=topk,gemv", + "SGLANG_TP_SIZE=2", + "SGLANG_HTTP_PORT=30000", + "SGLANG_CONTEXT_LEN=262144", + "SGLANG_MEM_FRACTION=0.85", + "SGLANG_CHUNKED_PREFILL=4096", + "SGLANG_MAX_RUNNING=64", + "SGLANG_PAGE_SIZE=64", + "SGLANG_MODEL=/model", + "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1" + ], + "Cmd": [ + "/entry.sh" + ], + "Image": "rahulunair/sglang-xpu@sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", + "Volumes": null, + "WorkingDir": "/sgl-workspace/sglang", + "Entrypoint": [ + "bash" + ], + "Labels": { + "org.opencontainers.image.description": "SGLang serving Qwen3.8-27B AWQ-INT4 or FP8 on four Intel Arc Pro B70 GPUs, with MTP and DFlash2 speculative decoding through the checkpoint's native 262,144-token context.", + "org.opencontainers.image.title": "Qwen3.8-27B on Intel Arc Pro", + "org.opencontainers.image.version": "24.04", + "xpu.qwen38.base": "rahulunair/sglang-xpu:qwen3.8-27b-20260819", + "xpu.qwen38.built": "20260820", + "xpu.qwen38.dflash2_source": "e5a3e4d30fa7abda95bafd2d697f9f9c48566114" + } + }, + "NetworkSettings": { + "SandboxID": "03f3987afe75b41bf575db9f24465c03eadd18d0db38c1159b382d341eaa41a2", + "SandboxKey": "/var/run/docker/netns/03f3987afe75", + "Ports": { + "30000/tcp": null, + "30001/tcp": [ + { + "HostIp": "0.0.0.0", + "HostPort": "30001" + }, + { + "HostIp": "::", + "HostPort": "30001" + } + ], + "5558/tcp": [ + { + "HostIp": "0.0.0.0", + "HostPort": "5558" + }, + { + "HostIp": "::", + "HostPort": "5558" + } + ] + }, + "Networks": { + "bridge": { + "IPAMConfig": null, + "Links": null, + "Aliases": null, + "DriverOpts": null, + "GwPriority": 0, + "NetworkID": "079840671febd9bc1caa55e83e63558724adbe92299b1abb5777fe9fa72b5975", + "EndpointID": "ffa4f82708a4cfbd1ea62d5979fb28595208835dc0b1b2922cd2fde66b611b75", + "Gateway": "172.17.0.1", + "IPAddress": "172.17.0.3", + "MacAddress": "5a:ad:91:4a:45:d3", + "IPPrefixLen": 16, + "IPv6Gateway": "", + "GlobalIPv6Address": "", + "GlobalIPv6PrefixLen": 0, + "DNSNames": null + } + } + }, + "ImageManifestDescriptor": { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "digest": "sha256:70aab3fb164f7941db3ec1b45be934bad70ff720f4013fd5b4429b3d90668e5a", + "size": 5653, + "platform": { + "architecture": "amd64", + "os": "linux" + } + } + } +] diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.key.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.key.json new file mode 100644 index 00000000..c8dd0d9d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.key.json @@ -0,0 +1 @@ +{"args": ["/entry.sh"], "binds": ["/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", "/dev/dri:/dev/dri", "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", "/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro", "/home/devcloud/hicache-shared:/hicache", "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro"], "devices": [{"CgroupPermissions": "rwm", "PathInContainer": "/dev/dri", "PathOnHost": "/dev/dri"}], "env": ["CCL_CONFIGURATION=cpu_gpu_dpcpp", "CCL_CONFIGURATION_PATH=", "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", "CTX=8192", "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", "DEBIAN_FRONTEND=noninteractive", "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", "DPLROOT=/opt/intel/oneapi/dpl/2022.13", "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", "KV_PORT=5558", "LANG=C.UTF-8", "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", "MKLROOT=/opt/intel/oneapi/mkl/2026.1", "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", "ONEAPI_ROOT=/opt/intel/oneapi", "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", "PORT=30001", "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", "SETVARS_COMPLETED=1", "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", "SGLANG_CHUNKED_PREFILL=4096", "SGLANG_CONTEXT_LEN=262144", "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", "SGLANG_HTTP_PORT=30000", "SGLANG_MAX_RUNNING=64", "SGLANG_MEM_FRACTION=0.85", "SGLANG_MODEL=/model", "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", "SGLANG_PAGE_SIZE=64", "SGLANG_TP_SIZE=2", "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", "TP_SIZE=2", "UMF_ROOT=/opt/intel/oneapi/umf/1.1", "UV_PYTHON_INSTALL_DIR=/opt/uv/python", "VIRTUAL_ENV=/opt/venv", "XPU_DENSE_INT8=1", "XPU_DENSE_INT8_MAX_M=8", "XPU_GDN_DECODE_BV=16", "XPU_KERNELS=topk,gemv", "XPU_MAMBA_PAGED=1", "XPU_MOE_ALIGN_SKIP=1", "XPU_MOE_BUILD=/opt/xpu/build-gemv", "XPU_REPO=/opt/xpu", "XPU_SYMM_AR=1", "XPU_SYMM_AR_MAX_BYTES=131072", "XPU_TOPK_BUILD=/opt/xpu/build", "XPU_UNROLL_SPEC_VERIFY=1", "ZE_AFFINITY_MASK=2,3"], "image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", "network": "bridge", "path": "bash", "ports": {"30001/tcp": [{"HostIp": "", "HostPort": "30001"}], "5558/tcp": [{"HostIp": "", "HostPort": "5558"}]}, "restart": {"MaximumRetryCount": 0, "Name": "no"}} diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh b/bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh new file mode 100755 index 00000000..7a51ba8a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh @@ -0,0 +1,304 @@ +#!/usr/bin/env bash +# prehold.sh STEP: H53's window steps on crescent (EXPERIMENTS.md, H53 amendment R1), run from the +# H53 checkout ~/k3dgpu-bench/k3d-gpu-h53 at the deploy commit. +# fetch models.sh qwen38 under the bench lock: both models at their pinned revisions, +# every LFS file against the Hub's sha256, B2's listing (prehold/fetch.log) +# setup crescent amendment E4: setup.sh under the bench lock (prehold/setup.txt) +# verify both models again against the Hub's sha256, and B2's listing against setup's +# (prehold/verify.txt) +# frontdoor TAG E2 item 7: a 16-token chat through Dynamo :8100 per serving pair, passing when it +# answers with content and that pair's SGLang num_requests_total rises (180 s each) +# stop open the window: `docker inspect` of both pairs into prehold/, then +# gpu-serving.sh stop for each (never rm), the cards' VRAM, window/open and the deadman +# smoke AG|B2 [RUNGS] +# a wiring smoke: solo-hold.sh's environment, except LIVE_PAIRS_GUARD=99999 (only the +# warm-up runs), LIVE_HOLD 1 (AG) or 2 (B2), B2's ladder RUNGS, output under smoke/H53/ +# podlog NAME a running smoke's pods, events and server logs (current and previous) into its dir +# holds FIRST LAST [RUNGS] +# the scored holds FIRST..LAST through solo-hold.sh, each on the four free cards and +# recorded from Prometheus; stops at the first hold that does not exit 0 (the window +# stays open), or once fewer than three blocks can still count (then restores); +# after hold 6, holds 7 and 8 run only when fewer than three blocks count; then `start` +# start close the window: gpu-serving.sh start for both pairs, the inspect comparison and +# frontdoor post +# status refresh the heartbeat; print the window's last events and the cards' VRAM +# deadman started by stop: restores both pairs once no window step is alive and none has +# run for 45 min (window/heartbeat), so a dead session cannot leave serving down +set -uo pipefail +B=${HOME}/k3dgpu-bench +REPO=${B}/k3d-gpu-h53 +C=${REPO}/bench/eatflow +R=${B}/results/H53 +O=${R}/prehold +W=${R}/window +S=${B}/smoke/H53 +mkdir -p "${O}" "${W}" +NODE_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c +PAIRS="qwen38-a-hr qwen38-b-hr" +# All four Arc Pro B70s: both pairs are stopped for the window (H53 pre-registration). +FREE="0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0" +PROM=http://127.0.0.1:9090 +STALE_S=2700 +FREE_GIB=2 +head_of() { (cd "${REPO}" && git rev-parse HEAD); } +ev() { echo "$(date -u +%FT%TZ) $*" | tee -a "${W}/events.log"; } +beat() { touch "${W}/heartbeat"; } +running() { docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null; } +alive() { [ -f "${W}/active.pid" ] && kill -0 "$(cat "${W}/active.pid")" 2>/dev/null; } +lock() { # lock: the bench lock on fd 9, as hold.sh takes it + exec 9> "${HOME}/k3dgpu-bench.lock" + flock -w 7200 9 || { echo "lock not taken in 7200 s"; exit 4; } +} +# vram: one " " line per card from Prometheus (xe-exporter's xe_vram_used_bytes). +vram() { + curl -s -m 10 "${PROM}/api/v1/query?query=xe_vram_used_bytes" | python3 -c ' +import json, sys +for r in sorted(json.load(sys.stdin)["data"]["result"], key=lambda r: r["metric"]["bdf"]): + print(r["metric"]["bdf"], round(int(r["value"][1]) / 2**30, 2))' +} +# ready_cards: the window is open, both pairs are stopped and every card holds under FREE_GIB. +ready_cards() { + local c v + [ -e "${W}/open" ] || { ev "window not open"; return 1; } + for c in ${PAIRS}; do + [ "$(running "${c}")" = false ] || { ev "${c} is running"; return 1; } + done + v=$(vram) || return 1 + [ "$(wc -l <<< "${v}")" -eq 4 ] || { ev "Prometheus lists $(wc -l <<< "${v}") cards, not 4"; return 1; } + if awk -v g="${FREE_GIB}" '$2 >= g {bad=1} END {exit !bad}' <<< "${v}"; then + ev "a card holds ${FREE_GIB} GiB or more: $(tr '\n' ';' <<< "${v}")"; return 1 + fi + ev "cards free: $(tr '\n' ';' <<< "${v}")" +} +# wait_cards: ready_cards, polled for up to 2 min (Prometheus scrapes every 15 s). +wait_cards() { + local _ + for _ in $(seq 24); do ready_cards > /dev/null 2>&1 && break; sleep 5; done + ready_cards +} +# prom T0 T1 OUT: per-card VRAM and engine utilisation over [T0, T1] (epoch s), 15 s steps. +prom() { + python3 - "$1" "$2" "$3" "${PROM}" <<'PY' +import json, sys, urllib.parse, urllib.request +t0, t1, out, prom = float(sys.argv[1]), float(sys.argv[2]), sys.argv[3], sys.argv[4] +res = {"t0": t0, "t1": t1} +for q in ("xe_vram_used_bytes", "xe_engine_utilization_percent"): + u = f"{prom}/api/v1/query_range?" + urllib.parse.urlencode({"query": q, "start": t0, "end": t1, "step": 15}) + with urllib.request.urlopen(u, timeout=30) as r: + res[q] = json.load(r)["data"]["result"] +peak = {} +for s in res["xe_vram_used_bytes"]: + peak[s["metric"]["bdf"]] = round(max(int(v) for _, v in s["values"]) / 2**30, 2) +res["vram_peak_gib"] = peak +busy = {} +for s in res["xe_engine_utilization_percent"]: + b = s["metric"].get("bdf", "?") + busy[b] = max(busy.get(b, 0.0), max(float(v) for _, v in s["values"])) +res["engine_peak_pct"] = {k: round(v, 1) for k, v in sorted(busy.items())} +with open(out, "w") as f: + json.dump(res, f) +print("vram peak GiB", peak, "engine peak %", res["engine_peak_pct"]) +PY +} +frontdoor() { + local f=${O}/frontdoor-${1:-check}.txt P b end ok c + : > "${f}" + n() { curl -s -m 8 "http://127.0.0.1:$1/metrics" | awk '/^sglang:num_requests_total/{s+=$2} END{print s+0}'; } + for P in 30000 30001; do + b=$(n "${P}") end=$(( SECONDS + 180 )) ok=0 c="" + until [ "${SECONDS}" -ge "${end}" ]; do + c=$(curl -s -m 60 http://127.0.0.1:8100/v1/chat/completions -H 'Content-Type: application/json' \ + -d '{"model":"Qwen3.8-27B","messages":[{"role":"user","content":"Say OK"}],"max_tokens":16,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}' \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["choices"][0]["message"]["content"].strip())' 2>/dev/null) + if [ -n "${c}" ] && [ "$(n "${P}")" -gt "${b}" ]; then ok=1; break; fi + sleep 5 + done + echo "$(date -u +%FT%TZ) :${P} pass=${ok} before=${b} after=$(n "${P}") content=${c:-none}" | tee -a "${f}" + done +} +# The parts of `docker inspect` a restart must keep: image, command, env, devices, mounts, ports. +inspect_key() { + docker inspect "$1" | python3 -c ' +import json, sys +d = json.load(sys.stdin)[0] +h = d["HostConfig"] +print(json.dumps({"image": d["Image"], "path": d["Path"], "args": d["Args"], "env": sorted(d["Config"]["Env"]), + "devices": h.get("Devices"), "binds": h.get("Binds"), "network": h.get("NetworkMode"), + "ports": h.get("PortBindings"), "restart": h.get("RestartPolicy")}, sort_keys=True))' +} +restore() { + local c p pids=() rc=0 + rm -f "${W}/open" + for c in ${PAIRS}; do + if [ "$(running "${c}")" = true ]; then ev "${c} already running"; continue; fi + bash "${C}/crescent/gpu-serving.sh" start "${c}" >> "${W}/restore.log" 2>&1 & + pids+=($!) + done + for p in "${pids[@]}"; do wait "${p}" || rc=1; done + for c in ${PAIRS}; do + ev "${c} running=$(running "${c}"); $(bash "${C}/crescent/gpu-serving.sh" check "${c}" 2>&1 | head -1)" + inspect_key "${c}" > "${O}/inspect-${c}.after.key.json" + if cmp -s "${O}/inspect-${c}.before.key.json" "${O}/inspect-${c}.after.key.json"; then + ev "${c}: image, command, env, devices, mounts and ports as before the stop" + else + ev "${c}: inspect DIFFERS from before the stop"; rc=1 + fi + done + frontdoor post + ev "window closed (restore rc=${rc})" + return "${rc}" +} +blocks() { # the number of blocks that count so far (solo-report's block_level.n_blocks, on a copy) + local t + t=$(mktemp -d) + cp "${R}/pairs/registration.json" "${R}/pairs/attempts.jsonl" "${R}/pairs/solo.jsonl" "${t}/" 2>/dev/null + python3 "${C}/agentbench.py" solo-report "${t}" --hyp H53 --arm-a AG --arm-b B2 > /dev/null 2>&1 + python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["block_level"]["n_blocks"])' "${t}/analysis.json" 2>/dev/null || echo 0 + rm -rf "${t}" +} +alive_blocks() { # the blocks that can still count (solo.alive_blocks); a hold has run once hold.sh wrote its rc + local ran="" n + for n in 1 2 3 4 5 6 7 8; do + compgen -G "${B}/runs/H53-hold${n}-*/rc" > /dev/null && ran="${ran} ${n}" + done + python3 - "${C}" "${R}/pairs/solo.jsonl" "${ran}" <<'PY' +import os, sys +sys.path.insert(0, sys.argv[1]) +import pairs as PR, solo +rows = PR.read_jsonl(sys.argv[2]) if os.path.exists(sys.argv[2]) else [] +print(" ".join(map(str, solo.alive_blocks(rows, {int(x) for x in sys.argv[3].split()})))) +PY +} +one_hold() { # one_hold N RUNGS + local n=$1 arm t0 t1 rc + arm=$(python3 "${C}/pairs.py" arm H53 "${n}") || return 2 + beat + wait_cards || { ev "hold ${n}: cards not free, no hold"; return 6; } + t0=$(date +%s) + ev "hold ${n} (${arm}) start" + LIVE_GPU_FREE="${FREE}" LIVE_B2_LADDER="$2" bash "${C}/crescent/solo-hold.sh" "${n}" "${arm}" + rc=$? + t1=$(date +%s) + beat + mkdir -p "${R}/hold${n}" + prom "${t0}" "${t1}" "${R}/hold${n}/xe-prom.json" >> "${W}/events.log" 2>&1 + ev "hold ${n} (${arm}) rc=${rc} ($(( t1 - t0 )) s); $(grep -h 'complete attempt\|ABORTED\|NOT DELIVERABLE' "${R}/hold${n}/live.log" 2>/dev/null | tail -1)" + return "${rc}" +} +case ${1:?usage: prehold.sh fetch|setup|verify|frontdoor|stop|smoke|podlog|holds|start|status|deadman} in +fetch) + lock + { echo "lock-taken $(date -u +%FT%TZ)"; head_of + HF_HUB_DISABLE_PROGRESS_BARS=1 bash "${C}/crescent/models.sh" qwen38 + echo "rc=$? $(date -u +%FT%TZ)"; } > "${O}/fetch.log" 2>&1 ;; +setup) + lock + { echo "lock-taken $(date -u +%FT%TZ)"; head_of + bash "${C}/crescent/setup.sh" + echo "rc=$? $(date -u +%FT%TZ)" + (cd "${B}/bin" && sha256sum k3d kubectl helm uv); } > "${O}/setup.txt" 2>&1 + tail -12 "${O}/setup.txt" ;; +verify) + lock + { echo "lock-taken $(date -u +%FT%TZ)"; head_of + bash "${C}/crescent/models.sh" verify "${B}/models/Qwen3.8-27B" Qwen/Qwen3.8-27B 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 + echo "Qwen3.8-27B rc=$?" + bash "${C}/crescent/models.sh" verify "${B}/models/Qwen3.8-27B-int4-ov" OpenVINO/Qwen3.8-27B-int4-ov 7b20ea8c7df7223a85ff27bab5c473f0cb038943 + echo "Qwen3.8-27B-int4-ov rc=$?" + bash "${C}/crescent/listing.sh" "${B}/models/Qwen3.8-27B" | cmp -s - "${B}/models/Qwen3.8-27B.listing" + echo "listing equal to setup's: rc=$?"; date -u +%FT%TZ; } > "${O}/verify.txt" 2>&1 + grep -E 'rc=|BAD|MISSING' "${O}/verify.txt" ;; +frontdoor) frontdoor "${2:-check}" ;; +stop) + [ ! -e "${W}/open" ] || { ev "window already open"; exit 1; } + for c in ${PAIRS}; do + docker inspect "${c}" > "${O}/inspect-${c}.before.json" || exit 1 + inspect_key "${c}" > "${O}/inspect-${c}.before.key.json" || exit 1 + done + ev "VRAM before the stop: $(vram | tr '\n' ';')" + date -u +%FT%TZ > "${W}/open" + beat + for c in ${PAIRS}; do + if ! bash "${C}/crescent/gpu-serving.sh" stop "${c}" >> "${W}/stop.log" 2>&1; then + ev "${c} NOT stopped (see stop.log): restoring"; restore; exit 1 + fi + ev "${c} stopped" + done + setsid nohup bash "$0" deadman > "${W}/deadman.log" 2>&1 < /dev/null & + echo $! > "${W}/deadman.pid" + ev "window open; deadman pid $(cat "${W}/deadman.pid")" + wait_cards ;; +smoke) + ARM=${2:?usage: prehold.sh smoke AG|B2 [RUNGS]} RUNGS=${3:-} + case ${ARM} in AG) H=1 ;; B2) H=2 ;; *) echo "smoke: ARM must be AG or B2"; exit 2 ;; esac + name=H53-smoke-${ARM}${RUNGS:+-${RUNGS}} + [ "${ARM}" = AG ] || name=H53-smoke-B2-${RUNGS:-a} + beat + wait_cards || exit 6 + mkdir -p "${S}/${name}/pairs" + echo $$ > "${W}/active.pid" + ev "smoke ${name} start" + exec "${C}/crescent/hold.sh" "${name}" 2400 env K3D_GPU_IMAGE="${NODE_IMAGE}" K3D_GPU_K3S_ARGS="--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1%" \ + LIVE_MODE=solo LIVE_SOLO="${ARM}" LIVE_CAP=2370 LIVE_HOLD="${H}" LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR="${S}/${name}/pairs" \ + LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE="${FREE}" LIVE_B2_TP=4 LIVE_B2_LADDER="${RUNGS}" \ + LIVE_EAT_XPU_CHART="${B}/eat-pr21/core/helm-charts/vllm" LIVE_MODELS="${B}/models" LIVE_BLUEPRINT="${B}/eab" \ + LIVE_PY="${B}/mafvenv/bin/python" PATH="${B}/bin:${PATH}" \ + bash "${C}/live.sh" H53 "${S}/${name}/hold" ;; +podlog) + d=${S}/${2:?usage: prehold.sh podlog NAME}/hold + export KUBECONFIG=${d}/kubeconfig PATH=${B}/bin:${PATH} + beat + kubectl get pods -A -o wide > "${d}/pods.txt" 2>&1 + kubectl get events -A --sort-by=.lastTimestamp > "${d}/events.txt" 2>&1 + for ns in default k3d-gpu-agent; do + for p in $(kubectl -n "${ns}" get pods -o name 2>/dev/null | grep -v gpu-hold); do + kubectl -n "${ns}" describe "${p}" > "${d}/describe-${p##*/}.txt" 2>&1 + kubectl -n "${ns}" logs "${p}" --all-containers > "${d}/log-${p##*/}.txt" 2>&1 + kubectl -n "${ns}" logs "${p}" --all-containers --previous > "${d}/log-${p##*/}.previous.txt" 2>&1 + done + done + ls -l "${d}" ;; +holds) + first=${2:?usage: prehold.sh holds FIRST LAST [RUNGS]} last=${3:?} rungs=${4:-} + echo $$ > "${W}/active.pid" + ev "holds ${first}-${last} (ladder '${rungs}') at $(head_of)" + n=${first} + while [ "${n}" -le "${last}" ]; do + one_hold "${n}" "${rungs}" || { ev "hold ${n} did not exit 0: holds stop, the window stays open"; exit 1; } + a=$(alive_blocks) + ev "after hold ${n}: blocks that can still count: ${a:-none}" + if [ "$(wc -w <<< "${a}")" -lt 3 ]; then + ev "fewer than three blocks can still count: no further hold (futility, R1 item 5)"; break + fi + if [ "${n}" -eq 6 ] && [ "${last}" -eq 6 ]; then + k=$(blocks) + ev "after hold 6: ${k} block(s) count" + [ "${k}" -ge 3 ] || { last=8; ev "fewer than three blocks count: holds 7 and 8 run"; } + fi + n=$(( n + 1 )) + done + restore ;; +start) beat; restore ;; +status) + beat + tail -n "${2:-8}" "${W}/events.log" + vram | tr '\n' ';'; echo ;; +deadman) + while [ -e "${W}/open" ]; do + sleep 60 + [ -e "${W}/open" ] || break + alive && continue + age=$(( $(date +%s) - $(stat -c %Y "${W}/heartbeat" 2>/dev/null || echo 0) )) + [ "${age}" -ge "${STALE_S}" ] || continue + # No hold of any row runs while serving comes back: take the bench lock first. + exec 9> "${HOME}/k3dgpu-bench.lock" + flock -w 7200 9 || { ev "deadman: lock not taken"; continue; } + if [ -e "${W}/open" ] && ! alive; then + ev "deadman: no window step for ${age} s: restoring serving" + restore + fi + flock -u 9 + break + done ;; +*) echo "usage: prehold.sh fetch|setup|verify|frontdoor|stop|smoke|podlog|holds|start|status|deadman"; exit 2 ;; +esac diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/setup.txt b/bench/eatflow/results/2026-10-20/H53/prehold/setup.txt new file mode 100644 index 00000000..87e23150 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/setup.txt @@ -0,0 +1,16 @@ +lock-taken 2026-10-30T00:13:39Z +1f39488be7695dc0d1848347bf1c91f04025bc55 +k3d: OK +kubectl: OK +helm.tgz: OK +uv.tgz: OK +k3d version v5.9.0 +k3s version v1.35.5-k3s1 (default) +Client Version: v1.37.2 +v4.3.1+gc0d08d0 +uv 0.13.6 (x86_64-unknown-linux-gnu) +rc=0 2026-10-30T00:13:42Z +06d8f25bc3a971c4eb29e0ff08429b180402db0f4dec838c9eac427e296800a0 k3d +c04487523e7aa7d1493f7808174c899db93d7a9cf57531420cd76b9bd415f8f4 kubectl +b66cb19f2b31f9d427632ea6c4b4a1ff89b74ecaeaee323cd166da41a7af53bd helm +23803452c0fbe125033054cd9ff245f1d649fb65b4b73f68429f4abda432436d uv diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/verify.txt b/bench/eatflow/results/2026-10-20/H53/prehold/verify.txt new file mode 100644 index 00000000..234aead3 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/verify.txt @@ -0,0 +1,64 @@ +lock-taken 2026-10-30T00:13:53Z +1f39488be7695dc0d1848347bf1c91f04025bc55 +ok size 1570 .gitattributes +ok size 11544 LICENSE +ok size 65012 README.md +ok size 8952 chat_template.jinja +ok size 4312 config.json +ok size 238 crc32.txt +ok size 202 generation_config.json +ok size 3353259 merges.txt +ok ba0ce20aae489ad196733da5064bcdf159a1fe84f53336648196e1ebb7751b1c model-00001-of-00018.safetensors +ok 06a148c01bfbe3faa14a5f184a7ff29a706f7ae1c8b2705d2058e26d17a001fb model-00002-of-00018.safetensors +ok 2e1bf62cbcd406eaa64b60d10353e1f0ef4039d0976e56f05cabe953454f9968 model-00003-of-00018.safetensors +ok 511e34063187882659753c4d93f3859f93c019fd438d8813071921c81d9a3f1a model-00004-of-00018.safetensors +ok 635cb53446dc74f219740fc59e18b774f877b803b9722e289ca62575a6efa701 model-00005-of-00018.safetensors +ok 0bc5214fac607f0e6cc92eec3789d4b8559410ef9fce66621ba8158e8410dae0 model-00006-of-00018.safetensors +ok 80b0c49033e9a0d5762562aa12f4acdb7f54da586f3d0110f28c48d91cf07892 model-00007-of-00018.safetensors +ok 7192c5b66185d3592927daabee1cc19e6f6e0ce75988ee20e824b624765fda79 model-00008-of-00018.safetensors +ok af3c48cc37af44f3db6ae0579baf019180d48d9c527caa0a1f03ff85813a56d8 model-00009-of-00018.safetensors +ok 163490a76f3bea3a40855b7efc04ce6d27afaf1a34f0bbde495b9491f76457c9 model-00010-of-00018.safetensors +ok 5f3ae1b948aeee39da77aec558e8236cd65fe4d7cb7686a76bb007acc563c6d8 model-00011-of-00018.safetensors +ok a3de1c7114677a8f5ac5c4892c90e8238ea5c1e2038c80e757dfc87c3902ca55 model-00012-of-00018.safetensors +ok 06ab79a41f74c9c5cb734816feb0c7fc364104b227165ee7391231e1155aa02a model-00013-of-00018.safetensors +ok 4138ed94603065ba884bbcadedb04d7718bb40117e85e6f5c6fc5b9c05b7a85b model-00014-of-00018.safetensors +ok 69224e27b9de4e7dbf6fc936c6eaae08447bda3b80a6c31a871ab451173afd22 model-00015-of-00018.safetensors +ok 73cb9a1089fb6155cb648609478d6633be8a5c7d9ca5a05bc8925ce8a553cefe model-00016-of-00018.safetensors +ok beb51f01056142ac4984bd800507b0dd0fd18de57f8e9ef6ea41d1a3598983a8 model-00017-of-00018.safetensors +ok 1d3479509e21494658f9b64d317f5ea8e55c4025d28c702d6c4d0b356ce8ea06 model-00018-of-00018.safetensors +ok size 112216 model.safetensors.index.json +ok size 390 preprocessor_config.json +ok 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3 tokenizer.json +ok size 17928 tokenizer_config.json +ok size 385 video_preprocessor_config.json +ok size 6722759 vocab.json +Qwen3.8-27B rc=0 +ok size 1570 .gitattributes +ok size 8321 README.md +ok size 8952 chat_template.jinja +ok size 3679 config.json +ok size 213 generation_config.json +ok size 1790 openvino_config.json +ok 48fc2c6a7a1056994982911eb9695826acef933c3d742364c56d16317a0b2a79 openvino_detokenizer.bin +ok size 16771 openvino_detokenizer.xml +ok 95aac72ac7286707c5e0fd9b120ea7f22c93f3ae17e7f1263442587d1d887d50 openvino_language_model.bin +ok size 9736672 openvino_language_model.xml +ok bc8ef47f252ad5a72be11e48ba6a23da50a67d1cef03396a7d902ca9aa1d5bc8 openvino_mtp_model.bin +ok size 210992 openvino_mtp_model.xml +ok e2099d0d83dcdfb00c378b51b50a52d28116aee71eb4f81cef0fe19ae37a085f openvino_text_embeddings_model.bin +ok size 5920 openvino_text_embeddings_model.xml +ok ca1ff37f580b558f0119b57824240adc92624d6d2ef6dd96ca2a24f4f1368c05 openvino_tokenizer.bin +ok size 35358 openvino_tokenizer.xml +ok 3792b6946c1f475aaac5d6f3960c7b912f612104e4f39ad68643b707370ed8e7 openvino_vision_embeddings_merger_model.bin +ok size 1330718 openvino_vision_embeddings_merger_model.xml +ok 5a8661b052abd07563d0f7148934d1fb77d5a842647151304287beb5dcee5541 openvino_vision_embeddings_model.bin +ok size 8610 openvino_vision_embeddings_model.xml +ok 82ccc97be0351cd82ebe1717fd7fa9adf72a44982a7ced8e8a845d869325d9e5 openvino_vision_embeddings_pos_model.bin +ok size 5887 openvino_vision_embeddings_pos_model.xml +ok size 390 preprocessor_config.json +ok size 1300 processor_config.json +ok 87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4 tokenizer.json +ok size 1098 tokenizer_config.json +Qwen3.8-27B-int4-ov rc=0 +listing equal to setup's: rc=0 +2026-10-30T00:14:39Z diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-post.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-post.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-pre.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-pre.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/hold.log b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/hold.log new file mode 100644 index 00000000..5d3fe93c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/hold.log @@ -0,0 +1,41 @@ +== arm H53 start 00:15:14 (+0s) +Unable to find image 'python:3.14.8-slim@sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a' locally +docker.io/library/python@sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a: Pulling from library/python +82a60621211f: Pulling fs layer +f2be67225471: Pulling fs layer +77be27415a1b: Pulling fs layer +ac7730f4ce1f: Pulling fs layer +ac7730f4ce1f: Download complete +82a60621211f: Download complete +77be27415a1b: Download complete +c30c368ebede: Download complete +f2be67225471: Download complete +c3878e211f92: Download complete +f2be67225471: Pull complete +77be27415a1b: Pull complete +ac7730f4ce1f: Pull complete +82a60621211f: Pull complete +Digest: sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a +Status: Downloaded newer image for python@sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a +== k3d-gpu up 00:15:16 (+2s) +== wait-ready 00:18:24 (+190s) +Waiting for deployment "ovms" rollout to finish: 0 of 1 updated replicas are available... +deployment "ovms" successfully rolled out +== ready 00:19:15 (+241s) +ovms renderD130 0000:7c:00.0 +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 8090 after 0 ms: Could not connect to server +== load typical 00:19:31 (+257s) +== baseline typical 00:19:32 (+258s) +== solo hold 1 (AG), deadline +2250s 00:19:52 (+278s) +agentbench: AG: agent http://127.0.0.1:8090, proxy 127.0.0.1:18620 -> http://127.0.0.1:18640, pod cgroup /sys/fs/cgroup/system.slice/docker-0f3ecc5db6fc6a5011721ffe8eb1db36d348e0df866e428c31ae3e2c103f8c88.scope/kubepods/burstable/podaae2b796-aecd-4431-88ac-d979d4e06784 +agentbench: task 21: 1966 s left < 99999 s, hold ends +agentbench: solo hold 1 (AG): 0 complete attempt(s) +== finished 00:19:58 (+284s) +== cleanup 00:19:58 (+284s) +== done 00:20:00 (+286s) diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/orphans_killed b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/orphans_killed new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/orphans_killed @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/post.txt new file mode 100644 index 00000000..0c11eb8e --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/post.txt @@ -0,0 +1,15 @@ +2026-10-30T00:20:05Z + 00:20:05 up 44 days, 1:34, 10 users, load average: 8.61, 4.10, 3.05 +%CPU PID USER COMMAND +15.8 872705 65532 epp + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo + 0.5 2535 root containerd +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098586964 2538499400 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/pre.txt new file mode 100644 index 00000000..02e7dc9c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/pre.txt @@ -0,0 +1,15 @@ +2026-10-30T00:15:13Z + 00:15:13 up 44 days, 1:29, 10 users, load average: 5.09, 3.91, 2.91 +%CPU PID USER COMMAND +15.8 872705 65532 epp + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.6 1548406 root kworker/u384:1-events_power_efficient + 0.5 3882 10001 tempo +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098401352 2538685012 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/rc b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/rc new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/rc @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/serving.jsonl b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/serving.jsonl new file mode 100644 index 00000000..5c180b88 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/serving.jsonl @@ -0,0 +1,29 @@ +{"ts": 1793319324.14, "seconds": 10.0, "load1": 4.39, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.009, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.021, "xpumd": 0.002}, "serving_cpu": 0.095, "aiperf_procs": 0} +{"ts": 1793319334.168, "seconds": 10.028, "load1": 3.86, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.058, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.019, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.148, "aiperf_procs": 0} +{"ts": 1793319344.2, "seconds": 10.032, "load1": 3.27, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.008, "drm-exporter": 0.0, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319354.23, "seconds": 10.03, "load1": 3.08, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.008, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319364.258, "seconds": 10.028, "load1": 2.61, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319374.287, "seconds": 10.029, "load1": 2.21, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.007, "drm-exporter": 0.0, "loki": 0.022, "xpumd": 0.001}, "serving_cpu": 0.102, "aiperf_procs": 0} +{"ts": 1793319384.4, "seconds": 10.113, "load1": 1.87, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793319394.428, "seconds": 10.028, "load1": 1.58, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.08, "aiperf_procs": 0} +{"ts": 1793319404.457, "seconds": 10.028, "load1": 1.56, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.013, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.092, "aiperf_procs": 0} +{"ts": 1793319414.486, "seconds": 10.029, "load1": 1.32, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.011, "xpumd": 0.001}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793319424.518, "seconds": 10.032, "load1": 1.11, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.016, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.091, "aiperf_procs": 0} +{"ts": 1793319434.548, "seconds": 10.03, "load1": 0.94, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.022, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.098, "aiperf_procs": 0} +{"ts": 1793319444.643, "seconds": 10.095, "load1": 0.8, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.082, "aiperf_procs": 0} +{"ts": 1793319454.671, "seconds": 10.028, "load1": 0.67, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.006, "xpumd": 0.001}, "serving_cpu": 0.08, "aiperf_procs": 0} +{"ts": 1793319464.701, "seconds": 10.03, "load1": 0.57, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.001}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319474.734, "seconds": 10.033, "load1": 0.48, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793319484.766, "seconds": 10.032, "load1": 0.41, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.008, "otel-collector": 0.0, "tempo": 0.004, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.006, "xpumd": 0.003}, "serving_cpu": 0.08, "aiperf_procs": 0} +{"ts": 1793319494.796, "seconds": 10.03, "load1": 0.34, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.016, "xpumd": 0.002}, "serving_cpu": 0.098, "aiperf_procs": 0} +{"ts": 1793319504.918, "seconds": 10.122, "load1": 0.45, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793319514.946, "seconds": 10.028, "load1": 0.38, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.014, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793319524.976, "seconds": 10.03, "load1": 0.64, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.014, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.012, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.003}, "serving_cpu": 0.101, "aiperf_procs": 0} +{"ts": 1793319535.006, "seconds": 10.03, "load1": 0.69, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.091, "aiperf_procs": 0} +{"ts": 1793319545.038, "seconds": 10.032, "load1": 0.74, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.022, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.117, "aiperf_procs": 0} +{"ts": 1793319555.068, "seconds": 10.03, "load1": 1.01, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.013, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.027, "prometheus": 0.007, "alloy": 0.008, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.096, "aiperf_procs": 0} +{"ts": 1793319565.176, "seconds": 10.108, "load1": 1.01, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793319575.208, "seconds": 10.032, "load1": 7.01, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.036, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.027, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.116, "aiperf_procs": 0} +{"ts": 1793319585.244, "seconds": 10.036, "load1": 8.53, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.019, "grafana": 0.008, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.03, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.013, "xpumd": 0.002}, "serving_cpu": 0.11, "aiperf_procs": 0} +{"ts": 1793319595.278, "seconds": 10.034, "load1": 9.99, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.014, "grafana": 0.006, "otel-collector": 0.004, "tempo": 0.006, "node-exporter": 0.028, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.004}, "serving_cpu": 0.103, "aiperf_procs": 0} +{"ts": 1793319605.309, "seconds": 10.031, "load1": 8.61, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.009, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.029, "prometheus": 0.007, "alloy": 0.009, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.095, "aiperf_procs": 0} diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/stale_reaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/stale_reaped new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/stale_reaped @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/unreaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/unreaped new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/wait_s b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-post.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-post.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-pre.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-pre.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/hold.log b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/hold.log new file mode 100644 index 00000000..aba49025 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/hold.log @@ -0,0 +1,7 @@ +== arm H53 start 00:20:14 (+0s) +== k3d-gpu up 00:20:15 (+1s) +== wait-ready 00:23:06 (+172s) +Waiting for deployment "vllm-qwen38-27b-xpu" rollout to finish: 0 of 1 updated replicas are available... +== cleanup 00:25:19 (+305s) +== done 00:25:22 (+308s) +Terminated "$@" diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/orphans_killed b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/orphans_killed new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/orphans_killed @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/post.txt new file mode 100644 index 00000000..19586570 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/post.txt @@ -0,0 +1,15 @@ +2026-10-30T00:25:26Z + 00:25:26 up 44 days, 1:39, 11 users, load average: 0.80, 1.80, 2.32 +%CPU PID USER COMMAND +15.7 872705 65532 epp + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo + 0.5 2535 root containerd +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098591344 2538495020 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/pre.txt new file mode 100644 index 00000000..2da69f6c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/pre.txt @@ -0,0 +1,15 @@ +2026-10-30T00:20:14Z + 00:20:14 up 44 days, 1:34, 10 users, load average: 7.92, 4.03, 3.03 +%CPU PID USER COMMAND +15.8 872705 65532 epp + 4.0 1565186 devcloud bash + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098583812 2538502552 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/rc b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/rc new file mode 100644 index 00000000..fba7ed52 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/rc @@ -0,0 +1 @@ +143 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/serving.jsonl b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/serving.jsonl new file mode 100644 index 00000000..36ca245f --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/serving.jsonl @@ -0,0 +1,31 @@ +{"ts": 1793319624.968, "seconds": 10.0, "load1": 6.77, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.004}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793319634.997, "seconds": 10.029, "load1": 5.8, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.039, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.001}, "serving_cpu": 0.118, "aiperf_procs": 0} +{"ts": 1793319645.028, "seconds": 10.03, "load1": 4.91, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.008, "otel-collector": 0.001, "tempo": 0.022, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.014, "drm-exporter": 0.0, "loki": 0.013, "xpumd": 0.003}, "serving_cpu": 0.114, "aiperf_procs": 0} +{"ts": 1793319655.056, "seconds": 10.029, "load1": 4.8, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.008, "drm-exporter": 0.0, "loki": 0.024, "xpumd": 0.002}, "serving_cpu": 0.101, "aiperf_procs": 0} +{"ts": 1793319665.087, "seconds": 10.031, "load1": 3.8, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.017, "otel-collector": 0.0, "tempo": 0.004, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.095, "aiperf_procs": 0} +{"ts": 1793319675.122, "seconds": 10.034, "load1": 3.3, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793319685.231, "seconds": 10.109, "load1": 2.79, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.083, "aiperf_procs": 0} +{"ts": 1793319695.261, "seconds": 10.03, "load1": 2.36, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.079, "aiperf_procs": 0} +{"ts": 1793319705.293, "seconds": 10.033, "load1": 2.0, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.082, "aiperf_procs": 0} +{"ts": 1793319715.325, "seconds": 10.032, "load1": 1.69, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.001}, "serving_cpu": 0.091, "aiperf_procs": 0} +{"ts": 1793319725.354, "seconds": 10.029, "load1": 1.43, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.009, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793319735.387, "seconds": 10.033, "load1": 1.21, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.009, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793319745.498, "seconds": 10.111, "load1": 1.02, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319755.528, "seconds": 10.03, "load1": 0.86, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.021, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.096, "aiperf_procs": 0} +{"ts": 1793319765.555, "seconds": 10.027, "load1": 0.96, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.01, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319775.592, "seconds": 10.037, "load1": 0.81, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.016, "xpumd": 0.002}, "serving_cpu": 0.093, "aiperf_procs": 0} +{"ts": 1793319785.622, "seconds": 10.03, "load1": 0.69, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.017, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.01, "xpumd": 0.003}, "serving_cpu": 0.097, "aiperf_procs": 0} +{"ts": 1793319795.651, "seconds": 10.029, "load1": 0.65, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.085, "aiperf_procs": 0} +{"ts": 1793319805.754, "seconds": 10.103, "load1": 0.87, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.082, "aiperf_procs": 0} +{"ts": 1793319815.785, "seconds": 10.031, "load1": 0.74, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.006, "xpumd": 0.003}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793319825.816, "seconds": 10.031, "load1": 0.71, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793319835.848, "seconds": 10.032, "load1": 0.6, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793319845.88, "seconds": 10.032, "load1": 0.66, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793319855.915, "seconds": 10.035, "load1": 0.63, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.017, "node-exporter": 0.024, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.103, "aiperf_procs": 0} +{"ts": 1793319866.026, "seconds": 10.112, "load1": 0.61, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793319876.058, "seconds": 10.031, "load1": 0.82, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793319886.092, "seconds": 10.034, "load1": 0.93, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.016, "xpumd": 0.003}, "serving_cpu": 0.1, "aiperf_procs": 0} +{"ts": 1793319896.125, "seconds": 10.033, "load1": 0.94, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793319906.155, "seconds": 10.03, "load1": 0.79, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.017, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.011, "xpumd": 0.003}, "serving_cpu": 0.095, "aiperf_procs": 0} +{"ts": 1793319916.188, "seconds": 10.033, "load1": 0.67, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793319926.29, "seconds": 10.102, "load1": 0.8, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.009, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.021, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.081, "aiperf_procs": 0} diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/stale_reaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/stale_reaped new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/stale_reaped @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/unreaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/unreaped new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/wait_s b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-post.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-post.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-pre.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-pre.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/hold.log b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/hold.log new file mode 100644 index 00000000..72e77adc --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/hold.log @@ -0,0 +1,7 @@ +== arm H53 start 00:25:35 (+0s) +== k3d-gpu up 00:25:35 (+0s) +== wait-ready 00:28:27 (+172s) +Waiting for deployment "vllm-qwen38-27b-xpu" rollout to finish: 0 of 1 updated replicas are available... +== cleanup 00:36:21 (+646s) +== done 00:36:23 (+648s) +Terminated "$@" diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/orphans_killed b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/orphans_killed new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/orphans_killed @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/post.txt new file mode 100644 index 00000000..74fb4786 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/post.txt @@ -0,0 +1,15 @@ +2026-10-30T00:36:28Z + 00:36:28 up 44 days, 1:50, 11 users, load average: 5.38, 2.98, 2.37 +%CPU PID USER COMMAND +15.7 872705 65532 epp + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo + 0.5 2535 root containerd +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098604252 2538482112 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/pre.txt new file mode 100644 index 00000000..e2c3c3da --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/pre.txt @@ -0,0 +1,15 @@ +2026-10-30T00:25:35Z + 00:25:35 up 44 days, 1:40, 10 users, load average: 0.91, 1.79, 2.31 +%CPU PID USER COMMAND +15.7 872705 65532 epp + 3.8 1578915 devcloud bash + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098591488 2538494876 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/rc b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/rc new file mode 100644 index 00000000..fba7ed52 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/rc @@ -0,0 +1 @@ +143 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/serving.jsonl b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/serving.jsonl new file mode 100644 index 00000000..feb6fcbf --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/serving.jsonl @@ -0,0 +1,65 @@ +{"ts": 1793319945.496, "seconds": 10.0, "load1": 1.08, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.012, "aiperf010": 0.0, "xe-exporter": 0.055, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.018, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.15, "aiperf_procs": 0} +{"ts": 1793319955.527, "seconds": 10.031, "load1": 1.06, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.002}, "serving_cpu": 0.085, "aiperf_procs": 0} +{"ts": 1793319965.557, "seconds": 10.03, "load1": 1.05, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.009, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.008, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.083, "aiperf_procs": 0} +{"ts": 1793319975.587, "seconds": 10.03, "load1": 0.96, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.008, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793319985.616, "seconds": 10.029, "load1": 0.82, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319995.654, "seconds": 10.038, "load1": 0.76, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.006, "xpumd": 0.002}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793320005.756, "seconds": 10.102, "load1": 0.65, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.011, "drm-exporter": 0.0, "loki": 0.029, "xpumd": 0.001}, "serving_cpu": 0.111, "aiperf_procs": 0} +{"ts": 1793320015.785, "seconds": 10.029, "load1": 0.55, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793320025.818, "seconds": 10.032, "load1": 0.46, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.017, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320035.847, "seconds": 10.029, "load1": 0.39, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793320045.878, "seconds": 10.031, "load1": 0.33, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.078, "aiperf_procs": 0} +{"ts": 1793320055.906, "seconds": 10.028, "load1": 0.28, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793320066.005, "seconds": 10.098, "load1": 0.24, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.012, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793320076.034, "seconds": 10.03, "load1": 0.36, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.011, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320086.062, "seconds": 10.028, "load1": 0.38, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.079, "aiperf_procs": 0} +{"ts": 1793320096.103, "seconds": 10.041, "load1": 0.32, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320106.134, "seconds": 10.031, "load1": 0.27, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.078, "aiperf_procs": 0} +{"ts": 1793320116.162, "seconds": 10.028, "load1": 0.54, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793320126.281, "seconds": 10.119, "load1": 0.46, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.021, "xpumd": 0.003}, "serving_cpu": 0.105, "aiperf_procs": 0} +{"ts": 1793320136.311, "seconds": 10.03, "load1": 0.47, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.083, "aiperf_procs": 0} +{"ts": 1793320146.342, "seconds": 10.031, "load1": 0.78, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.021, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.112, "aiperf_procs": 0} +{"ts": 1793320156.373, "seconds": 10.031, "load1": 0.89, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320166.407, "seconds": 10.034, "load1": 0.98, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.085, "aiperf_procs": 0} +{"ts": 1793320176.438, "seconds": 10.031, "load1": 1.13, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.085, "aiperf_procs": 0} +{"ts": 1793320186.539, "seconds": 10.101, "load1": 1.19, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.003}, "serving_cpu": 0.092, "aiperf_procs": 0} +{"ts": 1793320196.573, "seconds": 10.035, "load1": 1.08, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.083, "aiperf_procs": 0} +{"ts": 1793320206.605, "seconds": 10.032, "load1": 1.21, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320216.639, "seconds": 10.034, "load1": 1.25, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.091, "aiperf_procs": 0} +{"ts": 1793320226.673, "seconds": 10.034, "load1": 1.21, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320236.708, "seconds": 10.034, "load1": 1.18, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.024, "xpumd": 0.003}, "serving_cpu": 0.103, "aiperf_procs": 0} +{"ts": 1793320246.806, "seconds": 10.098, "load1": 1.3, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.008, "otel-collector": 0.001, "tempo": 0.024, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.002}, "serving_cpu": 0.115, "aiperf_procs": 0} +{"ts": 1793320256.838, "seconds": 10.032, "load1": 1.34, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320266.87, "seconds": 10.032, "load1": 1.29, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.097, "aiperf_procs": 0} +{"ts": 1793320276.902, "seconds": 10.031, "load1": 1.46, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793320286.931, "seconds": 10.029, "load1": 1.7, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320296.961, "seconds": 10.03, "load1": 1.59, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320307.07, "seconds": 10.109, "load1": 1.5, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.003}, "serving_cpu": 0.094, "aiperf_procs": 0} +{"ts": 1793320317.102, "seconds": 10.032, "load1": 1.42, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.013, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320327.133, "seconds": 10.031, "load1": 1.36, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.092, "aiperf_procs": 0} +{"ts": 1793320337.165, "seconds": 10.031, "load1": 1.3, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320347.196, "seconds": 10.031, "load1": 1.33, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320357.226, "seconds": 10.03, "load1": 1.28, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.003}, "serving_cpu": 0.096, "aiperf_procs": 0} +{"ts": 1793320367.331, "seconds": 10.105, "load1": 1.24, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.012, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.002}, "serving_cpu": 0.096, "aiperf_procs": 0} +{"ts": 1793320377.366, "seconds": 10.035, "load1": 1.2, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793320387.399, "seconds": 10.034, "load1": 1.17, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.101, "aiperf_procs": 0} +{"ts": 1793320397.433, "seconds": 10.033, "load1": 1.22, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320407.461, "seconds": 10.029, "load1": 2.88, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320417.495, "seconds": 10.033, "load1": 2.59, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.013, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320427.598, "seconds": 10.104, "load1": 2.48, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.003}, "serving_cpu": 0.095, "aiperf_procs": 0} +{"ts": 1793320437.632, "seconds": 10.034, "load1": 2.1, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793320447.662, "seconds": 10.03, "load1": 6.98, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320457.692, "seconds": 10.03, "load1": 6.06, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.02, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.103, "aiperf_procs": 0} +{"ts": 1793320467.724, "seconds": 10.032, "load1": 5.28, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320477.753, "seconds": 10.029, "load1": 4.62, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.016, "xpumd": 0.002}, "serving_cpu": 0.101, "aiperf_procs": 0} +{"ts": 1793320487.862, "seconds": 10.109, "load1": 4.22, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.003}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793320497.893, "seconds": 10.031, "load1": 3.73, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.012, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320507.924, "seconds": 10.031, "load1": 3.38, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.1, "aiperf_procs": 0} +{"ts": 1793320517.953, "seconds": 10.029, "load1": 3.17, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320527.985, "seconds": 10.032, "load1": 2.84, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320538.016, "seconds": 10.031, "load1": 2.47, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320548.11, "seconds": 10.094, "load1": 2.17, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320558.141, "seconds": 10.031, "load1": 1.84, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.011, "xpumd": 0.001}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793320568.173, "seconds": 10.032, "load1": 1.63, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.009, "otel-collector": 0.001, "tempo": 0.019, "node-exporter": 0.024, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.106, "aiperf_procs": 0} +{"ts": 1793320578.204, "seconds": 10.03, "load1": 6.18, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793320588.237, "seconds": 10.033, "load1": 5.38, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.085, "aiperf_procs": 0} diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/stale_reaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/stale_reaped new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/stale_reaped @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/unreaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/unreaped new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/wait_s b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.err b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.err new file mode 100644 index 00000000..8ba33fa5 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.err @@ -0,0 +1,2 @@ +==> fit: OpenVINO/Qwen3.8-27B-int4-ov on 1 GPU via ovms, context 32768 x 6 seqs, GPU 28.5 GiB/card, host 16.6 GiB (pod limit 19Gi), host reserve 50.3 GiB +==> agent: OpenVINO/Qwen3.8-27B-int4-ov as 'qwen3.8-27b' on GPU (gpu.intel.com/xe), 2 CPU(s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.log new file mode 100644 index 00000000..75d2c8d6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.log @@ -0,0 +1,48 @@ +Langfuse not configured: set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY to enable tracing +Resolved MCP client class: agent_framework.MCPStreamableHTTPTool +Banking MCP client configured for http://127.0.0.1:18600/mcp +============================================================ +Banking Assistant with DevUI +============================================================ +Starting web interface at http://localhost:8090 + +Features: + • Natural-language banking queries + • Banking tools via MCP server: http://127.0.0.1:18600/mcp + • Persistent memory via Redis (text-only) + • OpenAI-compatible API (GenAI Gateway / LiteLLM) + +Prerequisites: + 1. Banking MCP server reachable at MCP_SERVER_URL + 2. Redis: docker run -d -p 6379:6379 redis/redis-stack-server + 3. Configure .env with your LLM endpoint + +============================================================ +Registering 1 in-memory entities +Authentication middleware enabled +Starting Agent Framework DevUI on 127.0.0.1:8090 +INFO: Started server process [1563224] +INFO: Waiting for application startup. +Starting Agent Framework Server +Initializing Agent Framework executor... +Set up TracerProvider for instrumentation +Enabled Agent Framework observability +No Agent Framework entities directory configured +Discovered 0 entities from directory +Registered in-memory entity: agent_in_memory_banking-assistant_359c4828784a45298ac97575b50b5a42 +Total entities available: 1 +OpenAI proxy mode available (OPENAI_API_KEY configured) +INFO: Application startup complete. +INFO: Uvicorn running on http://127.0.0.1:8090 (Press CTRL+C to quit) +INFO: 127.0.0.1:54408 - "GET /v1/entities HTTP/1.1" 200 OK +INFO: 127.0.0.1:55396 - "GET /v1/entities HTTP/1.1" 200 OK +Extracted entity_id: agent_in_memory_banking-assistant_359c4828784a45298ac97575b50b5a42 +Found entity: banking-assistant (agent) +Executing agent: agent_in_memory_banking-assistant_359c4828784a45298ac97575b50b5a42 +Negotiated protocol version: 2025-11-25 +HTTP Request: POST http://127.0.0.1:18620/v1/chat/completions "HTTP/1.1 200 OK" +Function name: get_accounts +Function get_accounts succeeded. +Function duration: 0.004757s +HTTP Request: POST http://127.0.0.1:18620/v1/chat/completions "HTTP/1.1 200 OK" +INFO: 127.0.0.1:55398 - "POST /v1/responses HTTP/1.1" 200 OK diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.sh b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.sh new file mode 100755 index 00000000..01e617f9 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# (Re)start the banking agent and wait for its DevUI. +[ -f '/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.pid' ] && kill "$(cat '/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.pid')" 2>/dev/null +for _ in $(seq 100); do (exec 3<>/dev/tcp/127.0.0.1/8090) 2>/dev/null || break; sleep 0.1; done +cd '/home/devcloud/k3dgpu-bench/eab/usecases/banking-assistant/src' && (OPENAI_BASE_URL=http://127.0.0.1:18620/v1 OPENAI_API_KEY=not-needed MODEL_NAME='Qwen/Qwen3.8-27B' \ + MCP_SERVER_URL=http://127.0.0.1:18600/mcp ENABLE_MEMORY=false AUTO_OPEN_BROWSER=false \ + DEVUI_HOST=127.0.0.1 exec '/home/devcloud/k3dgpu-bench/mafvenv/bin/python' main.py >> '/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.log' 2>&1 & + echo $! > '/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.pid') +for _ in $(seq 45); do + curl -fsS -o /dev/null --max-time 5 -H "Authorization: Bearer ${DEVUI_AUTH_TOKEN}" http://127.0.0.1:8090/v1/entities && exit 0 + sleep 2 +done +exit 1 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.yaml new file mode 100644 index 00000000..1faad8cc --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.yaml @@ -0,0 +1,112 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: k3d-gpu-agent +--- +# Models and the compiled-model cache live here, so restarts never re-download. +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ovms-models + namespace: k3d-gpu-agent +spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 20Gi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ovms + namespace: k3d-gpu-agent + labels: + app: ovms +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: ovms + template: + metadata: + labels: + app: ovms + annotations: + k3d-gpu.io/agent-device: GPU + k3d-gpu.io/agent-fit: "gpu 6x32768" + spec: + securityContext: + runAsUser: 5000 + runAsGroup: 5000 + fsGroup: 5000 + supplementalGroups: [990] + nodeSelector: + gpu.intel.com/xmx: "true" + initContainers: + # --pull downloads the model once and writes its graph.pbtxt (device, + # parsers). On later starts the files are already there and only the + # graph is rewritten, so a device change takes effect with no download. + - name: pull + image: openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --pull + - --source_model=OpenVINO/Qwen3.8-27B-int4-ov + - --model_repository_path=/models + - --model_name=qwen3.8-27b + - --task=text_generation + - --target_device=GPU + - --tool_parser=qwen3coder + - --max_num_seqs=6 + - --cache_size=12 + volumeMounts: + - name: models + mountPath: /models + containers: + - name: ovms + image: openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --rest_port=8000 + - --model_path=/models/OpenVINO/Qwen3.8-27B-int4-ov + - --model_name=qwen3.8-27b + - --cache_dir=/models/.ov_cache + - --log_level=INFO + ports: + - name: http + containerPort: 8000 + # No liveness probe: compiling a model for a GPU takes minutes. + readinessProbe: + httpGet: + path: /v2/health/ready + port: http + periodSeconds: 5 + resources: + requests: + cpu: "2" + memory: 19Gi + limits: + memory: 19Gi + gpu.intel.com/xe: 1 + volumeMounts: + - name: models + mountPath: /models + volumes: + - name: models + persistentVolumeClaim: + claimName: ovms-models +--- +apiVersion: v1 +kind: Service +metadata: + name: ovms + namespace: k3d-gpu-agent +spec: + selector: + app: ovms + ports: + - name: http + port: 8000 + targetPort: http diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/b-listing.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/b-listing.txt new file mode 100644 index 00000000..de902c4d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/b-listing.txt @@ -0,0 +1,34 @@ +.gitattributes 1570 1793310278.7959648510 +LICENSE 11544 1793310278.8089651410 +README.md 65012 1793310278.8199653860 +chat_template.jinja 8952 1793310278.8139652520 +config.json 4312 1793310278.8129652300 +crc32.txt 238 1793310278.8299656090 +generation_config.json 202 1793310278.8259655200 +merges.txt 3353259 1793310278.8589662550 +model-00001-of-00018.safetensors 3966730552 1793310507.8016561670 +model-00002-of-00018.safetensors 3043080328 1793310511.8507358520 +model-00003-of-00018.safetensors 2542796952 1793310459.7537105830 +model-00004-of-00018.safetensors 3988973152 1793310571.2509048500 +model-00005-of-00018.safetensors 2099339864 1793310430.8621419970 +model-00006-of-00018.safetensors 3979553696 1793310509.6426923980 +model-00007-of-00018.safetensors 2108759344 1793310436.7122571270 +model-00008-of-00018.safetensors 3979553696 1793310573.9919587940 +model-00009-of-00018.safetensors 2108759344 1793310591.7193076680 +model-00010-of-00018.safetensors 3979553696 1793310721.1158541950 +model-00011-of-00018.safetensors 2108759344 1793310632.3041063770 +model-00012-of-00018.safetensors 3979553696 1793310730.4580380490 +model-00013-of-00018.safetensors 2108759344 1793310660.3256578430 +model-00014-of-00018.safetensors 3979553696 1793310752.6844754650 +model-00015-of-00018.safetensors 2108759344 1793310664.6107421730 +model-00016-of-00018.safetensors 3979564040 1793310758.3635872300 +model-00017-of-00018.safetensors 2108759344 1793310689.8802394770 +model-00018-of-00018.safetensors 3392197344 1793310751.2064463780 +model.safetensors.index.json 112216 1793310660.4686606570 +preprocessor_config.json 390 1793310660.5806628610 +tokenizer.json 12809320 1793310661.7056850020 +tokenizer_config.json 17928 1793310661.8266873830 +video_preprocessor_config.json 385 1793310661.9656901190 +vocab.json 6722759 1793310662.3626979320 +77042094076611b69791a610065f28b7013b8c621795fa86ddccc8bac7d1b9df model.safetensors.index.json +191e0af232104ed8b65258cf3fb2b842e288008baca7633c11b82a1ac7203aab config.json diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/blueprint-commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/blueprint-commit.txt new file mode 100644 index 00000000..2cf3b989 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/blueprint-commit.txt @@ -0,0 +1 @@ +758d4845520ca5456bf732264ef02d92a26d25a6 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/commit.txt new file mode 100644 index 00000000..4af4a96a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/commit.txt @@ -0,0 +1 @@ +1f39488be7695dc0d1848347bf1c91f04025bc55 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/deploy.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/deploy.yaml new file mode 100644 index 00000000..33752ef7 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/deploy.yaml @@ -0,0 +1,98 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: k3d-gpu-agent +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ovms + namespace: k3d-gpu-agent + labels: + app: ovms +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: ovms + template: + metadata: + labels: + app: ovms + annotations: + k3d-gpu.io/agent-device: GPU + k3d-gpu.io/agent-fit: gpu 6x32768 + spec: + securityContext: + runAsUser: 5000 + runAsGroup: 5000 + fsGroup: 5000 + supplementalGroups: + - 990 + nodeSelector: + gpu.intel.com/xmx: 'true' + initContainers: + - name: pull + image: openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --pull + - --source_model=OpenVINO/Qwen3.8-27B-int4-ov + - --model_repository_path=/models + - --model_name=qwen3.8-27b + - --task=text_generation + - --target_device=GPU + - --tool_parser=qwen3coder + - --max_num_seqs=6 + - --cache_size=12 + volumeMounts: + - name: models + mountPath: /models + containers: + - name: ovms + image: openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --rest_port=8000 + - --model_path=/models/OpenVINO/Qwen3.8-27B-int4-ov + - --model_name=qwen3.8-27b + - --cache_dir=/models/.ov_cache + - --log_level=INFO + ports: + - name: http + containerPort: 8000 + readinessProbe: + httpGet: + path: /v2/health/ready + port: http + periodSeconds: 5 + resources: + requests: + cpu: '2' + memory: 19Gi + limits: + memory: 19Gi + gpu.intel.com/xe: 1 + volumeMounts: + - name: models + mountPath: /models + volumes: + - name: models + hostPath: + path: /var/apb-models + type: Directory +--- +apiVersion: v1 +kind: Service +metadata: + name: ovms + namespace: k3d-gpu-agent +spec: + selector: + app: ovms + ports: + - name: http + port: 8000 + targetPort: http diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-arms.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-arms.txt new file mode 100644 index 00000000..f44d9837 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-arms.txt @@ -0,0 +1 @@ +ovms renderD130 0000:7c:00.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-hold.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-hold.txt new file mode 100644 index 00000000..971e9a12 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-hold.txt @@ -0,0 +1,4 @@ +gpu-hold-1 renderD128 0000:12:00.0 +gpu-hold-2 renderD130 0000:7c:00.0 +gpu-hold-3 renderD129 0000:48:00.0 +gpu-hold-4 renderD131 0000:96:00.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/image.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/image.txt new file mode 100644 index 00000000..3fa78339 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/image.txt @@ -0,0 +1 @@ +aae2b796-aecd-4431-88ac-d979d4e06784 openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 docker.io/openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 \ No newline at end of file diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm-image.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm-image.txt new file mode 100644 index 00000000..89987653 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm-image.txt @@ -0,0 +1 @@ +sha256:73cbb215d3d43ae5b42816217b200386eb08d567767fcb36cba163152c5f3c79 docker.litellm.ai/berriai/litellm:v1.83.14-stable.patch.2@sha256:73cbb215d3d43ae5b42816217b200386eb08d567767fcb36cba163152c5f3c79 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm.yaml new file mode 100644 index 00000000..78b1e4d6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm.yaml @@ -0,0 +1,7 @@ +model_list: + - model_name: Qwen/Qwen3.8-27B + litellm_params: + model: openai/qwen3.8-27b + api_base: http://127.0.0.1:18630/v1 + api_key: not-needed + timeout: 600 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/live.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/live.log new file mode 100644 index 00000000..4d34134d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/live.log @@ -0,0 +1,13 @@ +== arm H53 start 00:15:14 (+0s) +== k3d-gpu up 00:15:16 (+2s) +== wait-ready 00:18:24 (+190s) +== ready 00:19:15 (+241s) +== load typical 00:19:31 (+257s) +== baseline typical 00:19:32 (+258s) +== solo hold 1 (AG), deadline +2250s 00:19:52 (+278s) +agentbench: AG: agent http://127.0.0.1:8090, proxy 127.0.0.1:18620 -> http://127.0.0.1:18640, pod cgroup /sys/fs/cgroup/system.slice/docker-0f3ecc5db6fc6a5011721ffe8eb1db36d348e0df866e428c31ae3e2c103f8c88.scope/kubepods/burstable/podaae2b796-aecd-4431-88ac-d979d4e06784 +agentbench: task 21: 1966 s left < 99999 s, hold ends +agentbench: solo hold 1 (AG): 0 complete attempt(s) +== finished 00:19:58 (+284s) +== cleanup 00:19:58 (+284s) +== done 00:20:00 (+286s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/load-start.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/load-start.log new file mode 100644 index 00000000..88743ad2 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/load-start.log @@ -0,0 +1 @@ +started typical on 24-47,72-95 (3 containers, ttl 2173s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/mcp.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/mcp.log new file mode 100644 index 00000000..aa260a00 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/mcp.log @@ -0,0 +1,30 @@ +Starting Banking MCP server on 127.0.0.1:18600 (path /mcp) +INFO: Started server process [1563213] +INFO: Waiting for application startup. +StreamableHTTP session manager started +INFO: Application startup complete. +INFO: Uvicorn running on http://127.0.0.1:18600 (Press CTRL+C to quit) +INFO: 127.0.0.1:45244 - "POST /mcp HTTP/1.1" 200 OK +Terminating session: None +INFO: 127.0.0.1:45256 - "POST /mcp HTTP/1.1" 202 Accepted +Terminating session: None +INFO: 127.0.0.1:45264 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type PingRequest +Terminating session: None +INFO: 127.0.0.1:45278 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type ListToolsRequest +Terminating session: None +INFO: 127.0.0.1:45288 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type PingRequest +Terminating session: None +INFO: 127.0.0.1:45302 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type ListPromptsRequest +Terminating session: None +INFO: 127.0.0.1:44382 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type CallToolRequest +Terminating session: None +INFO: Shutting down +INFO: Waiting for application shutdown. +StreamableHTTP session manager shutting down +INFO: Application shutdown complete. +INFO: Finished server process [1563213] diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/models.json b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/models.json new file mode 100644 index 00000000..f5aae991 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/models.json @@ -0,0 +1 @@ +{"data":[{"id":"qwen3.8-27b","object":"model","created":1793319555,"owned_by":"OVMS"}],"object":"list"} \ No newline at end of file diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/pf.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/pf.log new file mode 100644 index 00000000..40e3e1b8 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/pf.log @@ -0,0 +1,5 @@ +Forwarding from 127.0.0.1:18630 -> 8000 +Forwarding from [::1]:18630 -> 8000 +Handling connection for 18630 +Handling connection for 18630 +Handling connection for 18630 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/reset-mcp.sh b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/reset-mcp.sh new file mode 100755 index 00000000..01314a4c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/reset-mcp.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +kill "$(cat "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/mcp.pid")" 2>/dev/null +for _ in $(seq 50); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null || break; sleep 0.1; done +cd "/home/devcloud/k3dgpu-bench/eab/usecases/banking-assistant/src" && (MCP_HOST=127.0.0.1 MCP_PORT=18600 exec "/home/devcloud/k3dgpu-bench/mafvenv/bin/python" -m mcp_server.server >> "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/mcp.log" 2>&1 & + echo $! > "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/mcp.pid") +for _ in $(seq 120); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null && exit 0; sleep 0.5; done +exit 1 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/up.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/up.log new file mode 100644 index 00000000..ae57c60d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/up.log @@ -0,0 +1,33 @@ +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators: intel amx +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-e-h53' with image cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-e-h53' +INFO[0000] Created image volume k3d-k3dgpu-e-h53-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-e-h53-tools' +INFO[0001] Creating node 'k3d-k3dgpu-e-h53-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-e-h53' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-e-h53-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-e-h53' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-e-h53) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/kubeconfig +==> Waiting for nodes Ready... +node/k3d-k3dgpu-e-h53-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +==> intel device plugin auto-deployed by k3s. +==> Verifying accelerators are schedulable... +Waiting for daemon set "intel-gpu-plugin" rollout to finish: 0 of 1 updated pods are available... +daemon set "intel-gpu-plugin" successfully rolled out +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators schedulable: intel=4 amx=1. Try: k3d-gpu test diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/baseline-hold1.jsonl b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/baseline-hold1.jsonl new file mode 100644 index 00000000..0acd80b8 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/baseline-hold1.jsonl @@ -0,0 +1,2 @@ +{"ts": 1793319582.2942176, "tag": "baseline", "level": "typical", "seconds": 10.055, "cpus": "24-47,72-95", "busy_pct": 36.87, "load1": 7.97, "pkg_w": 188.04, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "baseline", "busy_cpus": "24-47,72-95", "load_expected": 3, "load_running": 3} +{"ts": 1793319592.4581883, "tag": "baseline", "level": "typical", "seconds": 10.056, "cpus": "24-47,72-95", "busy_pct": 36.84, "load1": 9.29, "pkg_w": 188.62, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "baseline", "busy_cpus": "24-47,72-95", "load_expected": 3, "load_running": 3} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/facts-hold1.json b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/facts-hold1.json new file mode 100644 index 00000000..faa13e7b --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/facts-hold1.json @@ -0,0 +1,349 @@ +{ + "argv": [ + "/home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/agentbench.py", + "solo", + "--hyp", + "H53", + "--hold", + "1", + "--arm", + "AG", + "--out", + "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs", + "--deadline", + "1793321564", + "--tasks", + "B24", + "--agent", + "http://127.0.0.1:8090", + "--upstream", + "http://127.0.0.1:18640", + "--proxy-port", + "18620", + "--restart", + "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.sh", + "--pod", + "/sys/fs/cgroup/system.slice/docker-0f3ecc5db6fc6a5011721ffe8eb1db36d348e0df866e428c31ae3e2c103f8c88.scope/kubepods/burstable/podaae2b796-aecd-4431-88ac-d979d4e06784", + "--mcp-url", + "http://127.0.0.1:18600/mcp", + "--reset", + "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/reset-mcp.sh", + "--sampling", + "agent_qwen38", + "--baseline", + "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs/baseline-hold1.jsonl", + "--level", + "typical", + "--guard-s", + "99999" + ], + "args": { + "cmd": "solo", + "hyp": "H53", + "hold": 1, + "arm": "AG", + "out": "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs", + "deadline": 1793321564.0, + "tasks": "B24", + "agent": "http://127.0.0.1:8090", + "upstream": "http://127.0.0.1:18640", + "proxy_port": 18620, + "restart": "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.sh", + "pod": "/sys/fs/cgroup/system.slice/docker-0f3ecc5db6fc6a5011721ffe8eb1db36d348e0df866e428c31ae3e2c103f8c88.scope/kubepods/burstable/podaae2b796-aecd-4431-88ac-d979d4e06784", + "entity": null, + "mcp_url": "http://127.0.0.1:18600/mcp", + "reset": "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/reset-mcp.sh", + "sampling": "agent_qwen38", + "cap_s": 240.0, + "guard_s": 99999.0, + "settle_s": 30.0, + "no_warmup": false, + "no_rapl": false, + "quiesce_s": 300.0, + "warmup_timeout_s": 600.0, + "baseline": "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs/baseline-hold1.jsonl", + "level": "typical", + "pre_s": 10.0 + }, + "host": { + "cpu": "Intel(R) Xeon(R) 6741P", + "kernel": "7.0.0-31-generic", + "loadavg": "9.29 3.97 2.99 22/5228 1564302", + "python": "3.14.4" + }, + "lock": { + "schema": "k3d-gpu.agent-bench-lock/v1", + "read": "2026-10-17", + "note": "Every external piece of the agent benchmark, pinned. Image digests: Docker Hub tag API (hub.docker.com/v2/repositories//tags/) or the repo's own lock files; Hugging Face revisions: huggingface.co/api/models/; Ollama: sha256 of the registry manifest body (registry.ollama.ai/v2/library/qwen3/manifests/, Accept application/vnd.docker.distribution.manifest.v2+json).", + "suites": { + "bfcl_eval": { + "version": "2025.12.17", + "wheel": "bfcl_eval-2025.12.17-py3-none-any.whl", + "wheel_sha256": "8555bc9407a56682ceb7d969e87eb724f6b679deb0ef05114d9c6e786406b103", + "pypi": "https://pypi.org/project/bfcl-eval/2025.12.17/", + "gorilla_commit": "f7cf7359b7ac615a0b294831c5ba2bc95ee4a000", + "gorilla_commit_url": "https://github.com/ShishirPatil/gorilla/commit/f7cf7359b7ac615a0b294831c5ba2bc95ee4a000", + "install": "uv pip install --python /bin/python --require-hashes -r bench/eatflow/bfcl-constraints.txt (every package pinned with its sha256, generated by uv pip compile --generate-hashes; the bfcl-eval wheel hash is the one above)", + "note": "PyPI also has 2025.12.17.1 and 2026.3.23; do not float." + }, + "bfcl_board": { + "url": "https://gorilla.cs.berkeley.edu/data_overall.csv", + "sha256_on_read": "ede5081a24bc5ac7f9b3b09a52af6bcfcd35aa0f1709239fed979db77a8159a6", + "row": "Qwen3-4B-Instruct-2507 (FC)", + "values": { + "overall": 35.68, + "non_live_simple": 75.5, + "multiple": 93.5, + "parallel": 92.5, + "parallel_multiple": 90.0, + "live_simple": 79.07, + "live_multiple": 76.16, + "irrelevance": 84.93 + }, + "handler": "bfcl_eval/constants/model_config.py: 'Qwen/Qwen3-4B-Instruct-2507-FC' -> QwenFCHandler (local_inference, /v1/completions, BFCL applies the chat template)" + }, + "blueprints": { + "repo": "https://github.com/intel/enterprise-ai-blueprints", + "tag": "v1.0.0", + "commit": "758d4845520ca5456bf732264ef02d92a26d25a6", + "banking_data_py_sha256": "37b7b9c84af4395c370c4ac2d3ca063b0b4f9d0d2054c082013f9ea6e3e13916", + "banking_server_py_sha256": "1edb561a599a7879330e4e83b5005825ba4090d928401bafb3c0c66e083b8e78", + "coding_agent_py_sha256": "c30abfe159aa35d70b37572acd4b55a0a89d31187480add3c73a1cacc2de5991", + "agent_framework": "agent-framework-core==1.19.0, agent-framework-openai==1.14.4, agent-framework-devui==1.0.0b260918, mcp==1.30.0 (usecases/banking-assistant/src/requirements.txt)" + }, + "eat": { + "repo": "https://github.com/intel/enterprise-agent-toolkit", + "ref": "release-1.0.3-hotfix", + "commit": "0d3438d7119de44cafc5903f66665c0b623e3333" + }, + "coding_agent": { + "k8s_agent_sandbox": "1.0.7", + "wheel": "k8s_agent_sandbox-1.0.7-py3-none-any.whl", + "wheel_sha256": "1c0b92044b1612c5fb65b95e85558fd6633b4c64b6e1cfb624ef8c4ca49f5c5c", + "agent_sandbox": "v1.0.7 (share/agent-sandbox.yml, k3d-gpu sandbox up)", + "constraints": "bench/eatflow/coding-constraints.txt", + "install": "uv pip install --python /bin/python --require-hashes -r bench/eatflow/coding-constraints.txt (Python 3.12; every package with its sha256)", + "note": "H43 amendment W1: the blueprint at 758d484 pins k8s-agent-sandbox 0.4.6 (v1alpha1 claims), which agent-sandbox v1.0.7 does not serve; 1.0.7 runs behind bench/eatflow/coding/sitecustomize.py. The router is bench/eatflow/sbrouter.py on the proxy_python image." + } + }, + "models": { + "target": { + "repo": "Qwen/Qwen3-4B-Instruct-2507", + "revision": "cdbee75f17c01a7cc42f958dc650907174af0554" + }, + "awq": { + "repo": "cyankiwi/Qwen3-4B-Instruct-2507-AWQ-4bit", + "revision": "c46e2277ae3dee7d030f45c58987b291edc5c1d6", + "note": "third-party quant: confound recorded" + }, + "gguf": { + "repo": "unsloth/Qwen3-4B-Instruct-2507-GGUF", + "revision": "a06e946bb6b655725eafa393f4a9745d460374c9", + "file": "Qwen3-4B-Instruct-2507-Q4_K_M.gguf", + "lfs_sha256": "3605803b982cb64aead44f6c1b2ae36e3acdb41d8e46c8a94c6533bc4c67e597", + "bytes": 2497281120 + }, + "ollama": { + "tag": "qwen3:4b-instruct-2507-q4_K_M", + "manifest_sha256": "0edcdef34593eac1aa2be9c7d06c432dcf81945adca5eca2f27662c18f168ba0", + "model_blob": "sha256:85e4a5b7b8ef0e48af0e8658f5aaab9c2324c76c1641493f4d1e25fce54b18b9" + }, + "draft": { + "repo": "OpenVINO/Qwen3-0.6B-int4-ov", + "revision": "f864c6106efb6c7f7b4ef274a78a98e37210dddd" + }, + "openvino_int4": { + "note": "No OpenVINO int4 export of Qwen3-4B-Instruct-2507 is published; A1/A2 use a self-export (optimum-intel, NNCF int4 sym group 128) whose xml/bin sha256 is recorded in the run's facts.json." + } + }, + "images": { + "ovms": { + "ref": "openvino/model_server:2026.4.1-gpu@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58", + "source": "images.lock / bench/competitors.lock" + }, + "vllm_cpu_eat": { + "ref": "vllm/vllm-openai-cpu:v0.24.0@sha256:b0e6f8604413a8bfcca8836bb4b57b47f1aa653957e07f8903ece531309981a1", + "source": "integrations/eat/vllm-values.yaml; Docker Hub tag v0.24.0 digest" + }, + "vllm_cuda": { + "ref": "vllm/vllm-openai@sha256:a4a4c0437bf7240089da5f08aa370c4aee17ae5290f7a3b468825ee26c4c3a6b", + "source": "bench/apb/arm.sh; Docker Hub tag v0.31.0, linux/amd64 manifest (the tag's index digest is sha256:c1c9f6fd5c109ba7f0546a59f5b2f15fb87f64c77782e90a27b648b42a8e67c3)" + }, + "ollama": { + "ref": "ollama/ollama:0.40.5@sha256:5d3a2f25f9825628daba94dc0b1d421c7d57d6c03ae5faa9ba050e058bab8241", + "source": "bench/competitors.lock; Docker Hub tag 0.40.5 digest" + }, + "llamacpp": { + "ref": "ghcr.io/ggml-org/llama.cpp:server-b11706@sha256:f41d47eee678c02656eefc9ed8cce76cab584aa7032805eeef0e6639fcfda485", + "source": "bench/competitors.lock" + }, + "stress_ng": { + "ref": "ghcr.io/colinianking/stress-ng:72a9d0cae0f420a887991d4024d3101c4af7a4fc@sha256:2ea9a7c3828eda1b1ca3d0b0c5a669ec252f9caf97a6a33aedcea6b2a66d1cbe", + "source": "bench/load/load.py" + }, + "proxy_python": { + "ref": "python:3.14.8-slim@sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a", + "source": "docs-build/Dockerfile" + }, + "redis": { + "ref": "redis:8.10.2-alpine@sha256:9f8d2d789b9a1d1e3f283f979aa9cbf393373806472e1ccd460e2f9c869b833b", + "source": "H43 (amendment W1): the coding agent's memory store; Docker Hub tag 8.10.2-alpine index digest, read 2026-10-28; bench/eatflow/live.sh REDIS_IMAGE" + } + }, + "sampling": { + "_note": "Pinned by the recording proxy into every agent LLM request (it replaces every sampling key the agent sent), so no engine default applies. agent_local: greedy, every local arm. agent_remote: arm R through harvest, which forbids temperature 0 and logprobs; top_k 1 is greedy there; GLM-5.3 always reasons, so reasoning_effort low (the operator's known-good harvest client setting) keeps the 256-token cap for the answer. BFCL keeps its own temperature 0.001 (the board's setting).", + "agent_local": { + "temperature": 0, + "top_p": 1, + "max_tokens": 1024 + }, + "agent_remote": { + "top_k": 1, + "max_tokens": 256, + "reasoning_effort": "low" + }, + "agent_gemma4": { + "_source": "crescent amendment M1: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized @ f1e06dc5, README 'Sampling Parameters' and generation_config.json; thinking off: the template default (enable_thinking false) sent explicitly, as REFERENCE.md 1.1 gives it per request", + "temperature": 1.0, + "top_p": 0.95, + "top_k": 64, + "max_tokens": 1024, + "chat_template_kwargs": { + "enable_thinking": false + } + }, + "agent_qwen38": { + "_source": "H53: Qwen/Qwen3.8-27B @ 1d4bf0f2 README 'Best Practices', instruct (non-thinking) mode; min_p 0.0 and repetition_penalty 1.0 are the engines' defaults and are not sent; thinking off by the card's chat_template_kwargs", + "temperature": 0.7, + "top_p": 0.8, + "top_k": 20, + "presence_penalty": 1.5, + "max_tokens": 1024, + "chat_template_kwargs": { + "enable_thinking": false + } + } + }, + "hypotheses": { + "H40": { + "workload": "BFCL-S100 board", + "arms": [ + "A3ref" + ], + "levels": [ + "none" + ], + "order": "single" + }, + "H41": { + "workload": "BFCL-S100 server-fc", + "arms": [ + "A3ref", + "A1", + "A3", + "B1", + "C1", + "C2" + ], + "levels": [ + "typical" + ], + "order": "ABBA" + }, + "H42": { + "workload": "B24 N=1", + "arms": [ + "A1", + "A2", + "A3", + "B1", + "C1", + "C2" + ], + "levels": [ + "typical" + ], + "order": "ABBA" + }, + "H43": { + "workload": "C12 N=1", + "arms": [ + "A1", + "A3", + "B1" + ], + "levels": [ + "typical", + "heavy" + ], + "order": "ABBA" + }, + "H44": { + "workload": "B24 first 12 tasks N=1", + "arms": [ + "A1", + "B1" + ], + "levels": [ + "none", + "heavy" + ], + "order": "alternate" + }, + "H45": { + "workload": "B15-RO ramp N=1,2,4,8", + "arms": [ + "A1", + "A3", + "C1", + "C2" + ], + "levels": [ + "typical" + ], + "order": "ramp" + }, + "H46": { + "workload": "H42 blocks (energy) + one 120 s load-only control window per arm", + "arms": [ + "A1", + "A2", + "A3", + "B1", + "C1", + "C2" + ], + "levels": [ + "typical" + ], + "order": "control" + }, + "H47": { + "workload": "B24 N=1", + "arms": [ + "A4", + "A1" + ], + "levels": [ + "typical" + ], + "order": "ABBA", + "precondition": "H34 SUPPORTED" + }, + "H48": { + "workload": "4 B24 tasks + 8 S100 ids, <= 20 LLM requests", + "arms": [ + "R" + ], + "levels": [ + "none" + ], + "order": "single", + "max_requests": 20 + } + } + }, + "ts": 1793319592.5549145, + "hold": 1, + "arm": "AG" +} \ No newline at end of file diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/proxy-AG-hold1.jsonl b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/proxy-AG-hold1.jsonl new file mode 100644 index 00000000..77c131b2 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/proxy-AG-hold1.jsonl @@ -0,0 +1,2 @@ +{"ts_send": 1793319592.8474762, "path": "/v1/chat/completions", "stream": true, "model": "Qwen/Qwen3.8-27B", "last_user": "My user id is U1001. Which account ids do I have? Answer with the ids only.", "first_user": "My user id is U1001. Which account ids do I have? Answer with the ids only.", "user_msgs": 1, "tool_results": [], "sampling": {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5, "max_tokens": 1024, "chat_template_kwargs": {"enable_thinking": false}}, "tools": ["get_user", "find_user", "get_accounts", "get_account_balance", "get_balances", "get_transactions", "get_loans", "get_loan_summary", "transfer_funds", "pay_loan"], "status": 200, "error": null, "ttft_s": 2.946187, "e2e_s": 3.673176, "content": "", "tool_calls": [{"name": "get_accounts", "arguments": "{\"user_id\":\"U1001\"}", "valid": true, "reason": "ok"}], "tool_source": "structured", "tool_valid": true, "finish_reason": "tool_calls", "error_body": null, "usage": {"completion_tokens": 30, "prompt_tokens": 1929, "total_tokens": 1959}, "prompt_tokens": 1929, "completion_tokens": 30, "cached_tokens": null, "tag": null} +{"ts_send": 1793319596.531258, "path": "/v1/chat/completions", "stream": true, "model": "Qwen/Qwen3.8-27B", "last_user": "My user id is U1001. Which account ids do I have? Answer with the ids only.", "first_user": "My user id is U1001. Which account ids do I have? Answer with the ids only.", "user_msgs": 1, "tool_results": ["{\n \"user_id\": \"U1001\",\n \"count\": 2,\n \"accounts\": [\n {\n \"account_id\": \"ACC-1001-CHK\",\n \"user_id\": \"U1001\",\n \"type\": \"checking\",\n \"nickname\": \"Everyday Checking\",\n \"currency\": \"USD\",\n \"opened_date\": \"2016-03-12\",\n \"status\": \"active\"\n },\n {\n \"account_id\": \"ACC-1001-SAV\",\n \"user_id\": \"U1001\",\n \"type\": \"savings\",\n \"nickname\": \"Rainy Day Savings\",\n \"currency\": \"USD\",\n \"opened_date\": \"2017-01-05\",\n \"status\": \"active\"\n }\n ]\n}"], "sampling": {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5, "max_tokens": 1024, "chat_template_kwargs": {"enable_thinking": false}}, "tools": ["get_user", "find_user", "get_accounts", "get_account_balance", "get_balances", "get_transactions", "get_loans", "get_loan_summary", "transfer_funds", "pay_loan"], "status": 200, "error": null, "ttft_s": 0.370188, "e2e_s": 1.521517, "content": "ACC-1001-CHK, ACC-1001-SAV\n\n\nACC-1001-CHK, ACC-1001-SAV", "tool_calls": [], "tool_source": "none", "tool_valid": null, "finish_reason": "stop", "error_body": null, "usage": {"completion_tokens": 38, "prompt_tokens": 2185, "total_tokens": 2223, "completion_tokens_details": {"reasoning_tokens": 0}}, "prompt_tokens": 2185, "completion_tokens": 38, "cached_tokens": null, "tag": null} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/registration.json b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/registration.json new file mode 100644 index 00000000..94baebcd --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/registration.json @@ -0,0 +1 @@ +{"hyp": "H53", "rule": "L3", "seed": "H53-crescent-2026-10-20"} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/warmups.jsonl b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/warmups.jsonl new file mode 100644 index 00000000..a02039a4 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/warmups.jsonl @@ -0,0 +1 @@ +{"hold": 1, "arm": "AG", "seconds": 5.476, "error": null, "answer": "ACC-1001-CHK, ACC-1001-SAV\n\n\nACC-1001-CHK, ACC-1001-SAV"} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/b-listing.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/b-listing.txt new file mode 100644 index 00000000..de902c4d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/b-listing.txt @@ -0,0 +1,34 @@ +.gitattributes 1570 1793310278.7959648510 +LICENSE 11544 1793310278.8089651410 +README.md 65012 1793310278.8199653860 +chat_template.jinja 8952 1793310278.8139652520 +config.json 4312 1793310278.8129652300 +crc32.txt 238 1793310278.8299656090 +generation_config.json 202 1793310278.8259655200 +merges.txt 3353259 1793310278.8589662550 +model-00001-of-00018.safetensors 3966730552 1793310507.8016561670 +model-00002-of-00018.safetensors 3043080328 1793310511.8507358520 +model-00003-of-00018.safetensors 2542796952 1793310459.7537105830 +model-00004-of-00018.safetensors 3988973152 1793310571.2509048500 +model-00005-of-00018.safetensors 2099339864 1793310430.8621419970 +model-00006-of-00018.safetensors 3979553696 1793310509.6426923980 +model-00007-of-00018.safetensors 2108759344 1793310436.7122571270 +model-00008-of-00018.safetensors 3979553696 1793310573.9919587940 +model-00009-of-00018.safetensors 2108759344 1793310591.7193076680 +model-00010-of-00018.safetensors 3979553696 1793310721.1158541950 +model-00011-of-00018.safetensors 2108759344 1793310632.3041063770 +model-00012-of-00018.safetensors 3979553696 1793310730.4580380490 +model-00013-of-00018.safetensors 2108759344 1793310660.3256578430 +model-00014-of-00018.safetensors 3979553696 1793310752.6844754650 +model-00015-of-00018.safetensors 2108759344 1793310664.6107421730 +model-00016-of-00018.safetensors 3979564040 1793310758.3635872300 +model-00017-of-00018.safetensors 2108759344 1793310689.8802394770 +model-00018-of-00018.safetensors 3392197344 1793310751.2064463780 +model.safetensors.index.json 112216 1793310660.4686606570 +preprocessor_config.json 390 1793310660.5806628610 +tokenizer.json 12809320 1793310661.7056850020 +tokenizer_config.json 17928 1793310661.8266873830 +video_preprocessor_config.json 385 1793310661.9656901190 +vocab.json 6722759 1793310662.3626979320 +77042094076611b69791a610065f28b7013b8c621795fa86ddccc8bac7d1b9df model.safetensors.index.json +191e0af232104ed8b65258cf3fb2b842e288008baca7633c11b82a1ac7203aab config.json diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/blueprint-commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/blueprint-commit.txt new file mode 100644 index 00000000..2cf3b989 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/blueprint-commit.txt @@ -0,0 +1 @@ +758d4845520ca5456bf732264ef02d92a26d25a6 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/commit.txt new file mode 100644 index 00000000..4af4a96a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/commit.txt @@ -0,0 +1 @@ +1f39488be7695dc0d1848347bf1c91f04025bc55 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/deploy.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/deploy.yaml new file mode 100644 index 00000000..664aad4b --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/deploy.yaml @@ -0,0 +1,189 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: vllm-qwen38-27b-xpu-config + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +data: + http_proxy: '' + https_proxy: '' + no_proxy: '' + TORCHINDUCTOR_CACHE_DIR: /tmp + NUMBA_CACHE_DIR: /tmp + HF_HOME: /data + OUTLINES_CACHE_DIR: /tmp/.cache/outlines + VLLM_NO_USAGE_STATS: '1' + DO_NOT_TRACK: '1' + CCL_ALLREDUCE: ring + CCL_PROCESS_LAUNCHER: none + CCL_WORKER_AFFINITY: '' + CCL_WORKER_OFFLOAD: '0' + CCL_ZE_IPC_EXCHANGE: pidfd + HF_ENDPOINT: https://huggingface.co + HF_HUB_DISABLE_XET: '1' + HF_HUB_ENABLE_HF_TRANSFER: '0' + HOME: /tmp + LOGNAME: vllm + ONEAPI_DEVICE_SELECTOR: level_zero:* + SYCL_CACHE_PERSISTENT: '1' + TORCH_LLM_ALLREDUCE: '1' + TRITON_CACHE_DIR: /tmp/.triton + TRITON_HOME: /tmp/.triton + USER: vllm + VLLM_ALLOW_LONG_MAX_MODEL_LEN: '1' + VLLM_ATTENTION_BACKEND: TORCH_SDPA + VLLM_TARGET_DEVICE: xpu + VLLM_USE_V1: '1' + VLLM_WORKER_MULTIPROC_METHOD: spawn + XDG_CACHE_HOME: /tmp/.cache + ZES_ENABLE_SYSMAN: '1' + ZE_ENABLE_PCI_ID_DEVICE_ORDER: '1' + ZE_FLAT_DEVICE_HIERARCHY: FLAT + HF_HUB_OFFLINE: '1' +--- +apiVersion: v1 +kind: Service +metadata: + name: vllm-qwen38-27b-xpu-service + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 8000 + protocol: TCP + name: vllm + selector: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-qwen38-27b-xpu + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +spec: + selector: + matchLabels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + template: + metadata: + annotations: null + labels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + spec: + securityContext: + fsGroup: 1001 + supplementalGroups: + - 990 + - 44 + containers: + - name: vllm + envFrom: + - configMapRef: + name: vllm-qwen38-27b-xpu-config + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - SYS_NICE + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: false + runAsUser: 0 + seccompProfile: + type: RuntimeDefault + image: vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + imagePullPolicy: IfNotPresent + command: + - bash + - -c + - source /opt/intel/oneapi/setvars.sh --force && python3 -m vllm.entrypoints.openai.api_server + "$@" + - -- + args: + - --host + - 0.0.0.0 + - --dtype + - float16 + - --block-size + - '64' + - --gpu-memory-utilization + - '0.9' + - --enforce-eager + - --no-enable-prefix-caching + - --trust-remote-code + - --disable-sliding-window + - --enable-auto-tool-choice + - --tool-call-parser + - qwen3_xml + - --tensor-parallel-size + - '4' + - --model + - /data/Qwen3.8-27B + - --served-model-name + - Qwen/Qwen3.6-27B + - --port + - '8000' + volumeMounts: + - mountPath: /data + name: model-volume + - mountPath: /dev/shm + name: shm + - mountPath: /tmp + name: tmp + - mountPath: /root/.cache/vllm + name: vllm-cache + ports: + - name: http + containerPort: 8000 + protocol: TCP + readinessProbe: + failureThreshold: 200 + httpGet: + path: /health + port: http + initialDelaySeconds: 10 + periodSeconds: 5 + resources: + limits: + gpu.intel.com/xe: '4' + volumes: + - name: model-volume + hostPath: + path: /var/apb-ro + type: Directory + - name: shm + emptyDir: + medium: Memory + sizeLimit: 1Gi + - name: tmp + emptyDir: {} + - name: vllm-cache + emptyDir: {} + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: ei-inference-eligible + operator: In + values: + - 'true' diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/describe-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/describe-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt new file mode 100644 index 00000000..d223e436 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/describe-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt @@ -0,0 +1,119 @@ +Name: vllm-qwen38-27b-xpu-799dc85d7c-ng77g +Namespace: default +Priority: 0 +Service Account: default +Node: k3d-k3dgpu-e-h53-server-0/172.19.0.2 +Start Time: Fri, 30 Oct 2026 00:23:06 +0000 +Labels: app.kubernetes.io/instance=vllm-qwen38-27b-xpu + app.kubernetes.io/name=vllm + pod-template-hash=799dc85d7c +Annotations: +Status: Running +IP: 10.42.0.14 +IPs: + IP: 10.42.0.14 +Controlled By: ReplicaSet/vllm-qwen38-27b-xpu-799dc85d7c +Containers: + vllm: + Container ID: containerd://bf0eaf47310725fbdc8ba34fe3d4ed8108c6ae783b775c4d753d6fc589705d8f + Image: vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + Image ID: docker.io/vllm/vllm-openai-xpu@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + Port: 8000/TCP (http) + Host Port: 0/TCP (http) + SeccompProfile: RuntimeDefault + Command: + bash + -c + source /opt/intel/oneapi/setvars.sh --force && python3 -m vllm.entrypoints.openai.api_server "$@" + -- + Args: + --host + 0.0.0.0 + --dtype + float16 + --block-size + 64 + --gpu-memory-utilization + 0.9 + --enforce-eager + --no-enable-prefix-caching + --trust-remote-code + --disable-sliding-window + --enable-auto-tool-choice + --tool-call-parser + qwen3_xml + --tensor-parallel-size + 4 + --model + /data/Qwen3.8-27B + --served-model-name + Qwen/Qwen3.6-27B + --port + 8000 + State: Waiting + Reason: CrashLoopBackOff + Last State: Terminated + Reason: Error + Exit Code: 1 + Started: Fri, 30 Oct 2026 00:25:08 +0000 + Finished: Fri, 30 Oct 2026 00:25:08 +0000 + Ready: False + Restart Count: 2 + Limits: + gpu.intel.com/xe: 4 + Requests: + gpu.intel.com/xe: 4 + Readiness: http-get http://:http/health delay=10s timeout=1s period=5s successThreshold=1 failureThreshold=200 + Environment Variables from: + vllm-qwen38-27b-xpu-config ConfigMap Optional: false + Environment: + Mounts: + /data from model-volume (rw) + /dev/shm from shm (rw) + /root/.cache/vllm from vllm-cache (rw) + /tmp from tmp (rw) + /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jxpk9 (ro) +Conditions: + Type Status + PodReadyToStartContainers True + Initialized True + Ready False + ContainersReady False + PodScheduled True +Volumes: + model-volume: + Type: HostPath (bare host directory volume) + Path: /var/apb-ro + HostPathType: Directory + shm: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: Memory + SizeLimit: 1Gi + tmp: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: + vllm-cache: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: + kube-api-access-jxpk9: + Type: Projected (a volume that contains injected data from multiple sources) + TokenExpirationSeconds: 3607 + ConfigMapName: kube-root-ca.crt + Optional: false + DownwardAPI: true +QoS Class: BestEffort +Node-Selectors: +Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s + node.kubernetes.io/unreachable:NoExecute op=Exists for 300s +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 2m12s default-scheduler Successfully assigned default/vllm-qwen38-27b-xpu-799dc85d7c-ng77g to k3d-k3dgpu-e-h53-server-0 + Normal Pulling 2m12s kubelet spec.containers{vllm}: Pulling image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" + Normal Pulled 28s kubelet spec.containers{vllm}: Successfully pulled image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" in 1m43.931s (1m43.931s including waiting). Image size: 4143776505 bytes. + Normal Created 11s (x3 over 28s) kubelet spec.containers{vllm}: Container created + Normal Started 11s (x3 over 28s) kubelet spec.containers{vllm}: Container started + Normal Pulled 11s (x2 over 26s) kubelet spec.containers{vllm}: Container image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" already present on machine and can be accessed by the pod + Warning BackOff 10s (x4 over 25s) kubelet spec.containers{vllm}: Back-off restarting failed container vllm in pod vllm-qwen38-27b-xpu-799dc85d7c-ng77g_default(6077fde9-5952-450e-b869-3aa47c254b6a) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/eat.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/eat.yaml new file mode 100644 index 00000000..7c2d0407 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/eat.yaml @@ -0,0 +1,301 @@ +--- +# Source: vllm/templates/secret.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: Secret +metadata: + name: vllm-qwen38-27b-xpu-secret + labels: + app: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +type: Opaque +data: +--- +# Source: vllm/templates/configmap.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: ConfigMap +metadata: + name: vllm-qwen38-27b-xpu-config + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +data: + http_proxy: "" + https_proxy: "" + no_proxy: "" + TORCHINDUCTOR_CACHE_DIR: "/tmp" + NUMBA_CACHE_DIR: "/tmp" + HF_HOME: "/data" + OUTLINES_CACHE_DIR: "/tmp/.cache/outlines" + VLLM_NO_USAGE_STATS: "1" + DO_NOT_TRACK: "1" + CCL_ALLREDUCE: "ring" + CCL_PROCESS_LAUNCHER: "none" + CCL_WORKER_AFFINITY: "" + CCL_WORKER_OFFLOAD: "0" + CCL_ZE_IPC_EXCHANGE: "pidfd" + DO_NOT_TRACK: "1" + HF_ENDPOINT: "https://huggingface.co" + HF_HUB_DISABLE_XET: "1" + HF_HUB_ENABLE_HF_TRANSFER: "0" + HOME: "/tmp" + LOGNAME: "vllm" + ONEAPI_DEVICE_SELECTOR: "level_zero:*" + SYCL_CACHE_PERSISTENT: "1" + TORCH_LLM_ALLREDUCE: "1" + TRITON_CACHE_DIR: "/tmp/.triton" + TRITON_HOME: "/tmp/.triton" + USER: "vllm" + VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1" + VLLM_ATTENTION_BACKEND: "TORCH_SDPA" + VLLM_NO_USAGE_STATS: "1" + VLLM_TARGET_DEVICE: "xpu" + VLLM_USE_V1: "1" + VLLM_WORKER_MULTIPROC_METHOD: "spawn" + XDG_CACHE_HOME: "/tmp/.cache" + ZES_ENABLE_SYSMAN: "1" + ZE_ENABLE_PCI_ID_DEVICE_ORDER: "1" + ZE_FLAT_DEVICE_HIERARCHY: "FLAT" +--- +# Source: vllm/templates/pvc.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: vllm-qwen38-27b-xpu-pvc + namespace: default +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 250Gi # Adjust this according to model storage size +--- +# Source: vllm/templates/service.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: v1 +kind: Service +metadata: + name: vllm-qwen38-27b-xpu-service + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 8000 + protocol: TCP + name: vllm + selector: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu +--- +# Source: vllm/templates/deployment.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-qwen38-27b-xpu + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +spec: + selector: + matchLabels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + template: + metadata: + annotations: + labels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + spec: + # + # serviceAccountName: default + securityContext: + fsGroup: 1001 + supplementalGroups: + - 990 + - 44 + containers: + - name: vllm + envFrom: + - configMapRef: + name: vllm-qwen38-27b-xpu-config + - secretRef: + name: vllm-qwen38-27b-xpu-secret + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - SYS_NICE + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: false + runAsUser: 0 + seccompProfile: + type: RuntimeDefault + image: "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" + imagePullPolicy: IfNotPresent + command: + - bash + - -c + - source /opt/intel/oneapi/setvars.sh --force && export LD_LIBRARY_PATH=/host-gpu-libs:${LD_LIBRARY_PATH} + && python3 -m vllm.entrypoints.openai.api_server "$@" + - -- + # command: + # - /bin/bash + # - -c + # - | + # python3 -m vllm.entrypoints.openai.api_server --dtype float16 --model Qwen/Qwen3.6-27B --port 8000 --tensor-parallel-size 4 --block-size 64 --max-model-len 32768 --disable-log-requests + args: + - "--host" + - "0.0.0.0" + - "--dtype" + - "float16" + - "--block-size" + - "64" + - "--gpu-memory-utilization" + - "0.9" + - "--enforce-eager" + - "--no-enable-prefix-caching" + - "--trust-remote-code" + - "--disable-sliding-window" + - "--enable-auto-tool-choice" + - "--tool-call-parser" + - "qwen3_xml" + - "--tensor-parallel-size" + - "4" + - "--model" + - "Qwen/Qwen3.6-27B" + - --served-model-name + - "Qwen/Qwen3.6-27B" + - "--port" + - "8000" + volumeMounts: + - mountPath: /data + name: model-volume + - mountPath: /dev/shm + name: shm + - mountPath: /tmp + name: tmp + - mountPath: /root/.cache/vllm + name: vllm-cache + - mountPath: /host-gpu-libs + name: host-gpu-libs + readOnly: true + - mountPath: /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1 + name: host-gpu-libs + readOnly: true + subPath: libze_intel_gpu.so.1 + - mountPath: /usr/lib/x86_64-linux-gnu/libze_loader.so.1 + name: host-gpu-libs + readOnly: true + subPath: libze_loader.so.1 + - mountPath: /etc/OpenCL/vendors + name: host-opencl-vendors + readOnly: true + - mountPath: /usr/lib/x86_64-linux-gnu/intel-opencl + name: host-intel-opencl + readOnly: true + ports: + - name: http + containerPort: 8000 + protocol: TCP + readinessProbe: + failureThreshold: 98 + httpGet: + path: /health + port: http + initialDelaySeconds: 120 + periodSeconds: 30 + resources: + limits: + "gpu.intel.com/xe": "4" + + volumes: + - name: model-volume + persistentVolumeClaim: + claimName: vllm-qwen38-27b-xpu-pvc + - name: shm + emptyDir: + medium: Memory + sizeLimit: 1Gi + - name: tmp + emptyDir: {} + - name: vllm-cache + emptyDir: {} + - hostPath: + path: /usr/lib/x86_64-linux-gnu + type: Directory + name: host-gpu-libs + - hostPath: + path: /etc/OpenCL/vendors + type: Directory + name: host-opencl-vendors + - hostPath: + path: /usr/lib/x86_64-linux-gnu/intel-opencl + type: Directory + name: host-intel-opencl + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: ei-inference-eligible + operator: In + values: + - "true" + +--- +# Source: vllm/templates/apisixroutes.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +--- +# Source: vllm/templates/chat-template-configmap.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +--- +# Source: vllm/templates/ingress.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# Please edit the object below. Lines beginning with a '#' will be ignored, +# and an empty file will abort the edit. If an error occurs while saving this file will be +# reopened with the relevant failures. +--- +# Source: vllm/templates/ingress_eks.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +--- +# Source: vllm/templates/route.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +--- +# Source: vllm/templates/servicemonitor.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/events.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/events.txt new file mode 100644 index 00000000..527d14ff --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/events.txt @@ -0,0 +1,163 @@ +NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE +default 4m21s Normal Scheduled pod/gpu-hold-2 Successfully assigned default/gpu-hold-2 to k3d-k3dgpu-e-h53-server-0 +default 4m21s Normal Scheduled pod/gpu-hold-3 Successfully assigned default/gpu-hold-3 to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 4m47s Normal Scheduled pod/nfd-master-74b6cfb7df-m4whr Successfully assigned node-feature-discovery/nfd-master-74b6cfb7df-m4whr to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 4m47s Normal Scheduled pod/nfd-gc-7d6955597d-v5ns2 Successfully assigned node-feature-discovery/nfd-gc-7d6955597d-v5ns2 to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/local-path-provisioner-6858d854cf-8t662 Successfully assigned kube-system/local-path-provisioner-6858d854cf-8t662 to k3d-k3dgpu-e-h53-server-0 +default 2m12s Normal Scheduled pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Successfully assigned default/vllm-qwen38-27b-xpu-799dc85d7c-ng77g to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/intel-gpu-plugin-hxvpb Successfully assigned kube-system/intel-gpu-plugin-hxvpb to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/metrics-server-778c4649b4-l4d4j Successfully assigned kube-system/metrics-server-778c4649b4-l4d4j to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 4m47s Normal Scheduled pod/nfd-worker-87d64 Successfully assigned node-feature-discovery/nfd-worker-87d64 to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/helm-install-gateway-api-crd-gfhvl Successfully assigned kube-system/helm-install-gateway-api-crd-gfhvl to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/coredns-577d995dff-c89zz Successfully assigned kube-system/coredns-577d995dff-c89zz to k3d-k3dgpu-e-h53-server-0 +default 4m54s Normal Starting node/k3d-k3dgpu-e-h53-server-0 +default 4m21s Normal Scheduled pod/gpu-hold-4 Successfully assigned default/gpu-hold-4 to k3d-k3dgpu-e-h53-server-0 +default 4m21s Normal Scheduled pod/gpu-hold-1 Successfully assigned default/gpu-hold-1 to k3d-k3dgpu-e-h53-server-0 +default 4m54s Normal NodeHasNoDiskPressure node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasNoDiskPressure +default 4m54s Normal CertificateExpirationOK node/k3d-k3dgpu-e-h53-server-0 Node and Certificate Authority certificates managed by k3s are OK +default 4m54s Normal Starting node/k3d-k3dgpu-e-h53-server-0 Starting kubelet. +default 4m54s Warning InvalidDiskCapacity node/k3d-k3dgpu-e-h53-server-0 invalid capacity 0 on image filesystem +default 4m54s Normal NodeHasSufficientMemory node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasSufficientMemory +default 4m54s Normal NodeReady node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeReady +default 4m54s Normal NodeHasSufficientPID node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasSufficientPID +default 4m54s Normal NodeAllocatableEnforced node/k3d-k3dgpu-e-h53-server-0 Updated Node Allocatable limit across pods +default 4m52s Normal NodePasswordValidationComplete node/k3d-k3dgpu-e-h53-server-0 Deferred node password secret validation complete +kube-system 4m51s Normal AppliedManifest addon/gateway-api-crd Applied manifest at "/var/lib/rancher/k3s/server/manifests/gateway-api-crd.yaml" +kube-system 4m51s Normal ApplyingManifest addon/ccm Applying manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml" +kube-system 4m51s Normal AppliedManifest addon/ccm Applied manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml" +kube-system 4m51s Normal ApplyingManifest addon/gaudi-device-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/gaudi-device-plugin.yaml" +kube-system 4m51s Normal ApplyingManifest addon/coredns Applying manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml" +kube-system 4m51s Normal AppliedManifest addon/coredns Applied manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml" +kube-system 4m51s Normal ApplyingManifest addon/gateway-api-crd Applying manifest at "/var/lib/rancher/k3s/server/manifests/gateway-api-crd.yaml" +kube-system 4m50s Normal ApplyingManifest addon/metrics-apiservice Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" +kube-system 4m50s Normal ApplyingManifest addon/auth-delegator Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-qat-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-qat-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-qat-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-qat-plugin.yaml" +default 4m50s Normal Synced node/k3d-k3dgpu-e-h53-server-0 Node synced successfully +kube-system 4m50s Normal AppliedManifest addon/resource-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" +kube-system 4m50s Normal ApplyingManifest addon/local-storage Applying manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-npu-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-npu-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-iaa-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-iaa-plugin.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-iaa-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-iaa-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-gpu-wsl-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-wsl-plugin.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-gpu-wsl-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-wsl-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-gpu-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-plugin.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-gpu-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/local-storage Applied manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml" +kube-system 4m50s Normal ApplyingManifest addon/aggregated-metrics-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" +kube-system 4m50s Normal AppliedManifest addon/aggregated-metrics-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-npu-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-npu-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/auth-delegator Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" +kube-system 4m50s Normal ApplyingManifest addon/auth-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" +kube-system 4m50s Normal AppliedManifest addon/auth-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" +kube-system 4m50s Normal AppliedManifest addon/metrics-apiservice Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" +kube-system 4m50s Normal ApplyingManifest addon/metrics-server-deployment Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" +kube-system 4m50s Normal AppliedManifest addon/metrics-server-deployment Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" +kube-system 4m50s Normal DeletingManifest addon/intel-gpu-dra Deleting manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-dra.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-dsa-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-dsa-plugin.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-dsa-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-dsa-plugin.yaml" +kube-system 4m50s Normal ApplyJob helmchart/gateway-api-crd Applying HelmChart from https://%{KUBERNETES_API}%/static/charts/gateway-api-crd-1.6.103.tgz using Job kube-system/helm-install-gateway-api-crd +kube-system 4m50s Normal ApplyingManifest addon/metrics-server-service Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" +kube-system 4m50s Normal AppliedManifest addon/gaudi-device-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/gaudi-device-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/metrics-server-service Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" +kube-system 4m50s Normal ApplyingManifest addon/node-feature-discovery Applying manifest at "/var/lib/rancher/k3s/server/manifests/node-feature-discovery.yaml" +kube-system 4m50s Normal ApplyingManifest addon/resource-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" +kube-system 4m49s Normal Suspended job/helm-install-gateway-api-crd Job suspended +default 4m49s Normal RegisteredNode node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 event: Registered Node k3d-k3dgpu-e-h53-server-0 in Controller +kube-system 4m48s Normal SuccessfulCreate replicaset/metrics-server-778c4649b4 Created pod: metrics-server-778c4649b4-l4d4j +kube-system 4m48s Normal DeletingManifest addon/traefik Deleting manifest at "/var/lib/rancher/k3s/server/manifests/traefik.yaml" +kube-system 4m48s Normal ResumeJob helmchart/gateway-api-crd Resumed synced Job kube-system/helm-install-gateway-api-crd +kube-system 4m48s Normal SuccessfulCreate replicaset/coredns-577d995dff Created pod: coredns-577d995dff-c89zz +node-feature-discovery 4m48s Normal SuccessfulCreate daemonset/nfd-worker Created pod: nfd-worker-87d64 +node-feature-discovery 4m48s Warning FailedCreatePodSandBox pod/nfd-worker-87d64 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "665049125198f0eec35b75b738ea9f7e00faed2fa597064534ec212c60c89b91": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m48s Warning FailedCreatePodSandBox pod/helm-install-gateway-api-crd-gfhvl Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "37b3a49edc472b8ffa47c8e9cd3de8fbe51ecba1b2a89d20fe8e165ef54a6cdb": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 4m48s Normal ScalingReplicaSet deployment/nfd-master Scaled up replica set nfd-master-74b6cfb7df from 0 to 1 +node-feature-discovery 4m48s Normal SuccessfulCreate replicaset/nfd-master-74b6cfb7df Created pod: nfd-master-74b6cfb7df-m4whr +kube-system 4m48s Normal SuccessfulCreate replicaset/local-path-provisioner-6858d854cf Created pod: local-path-provisioner-6858d854cf-8t662 +node-feature-discovery 4m48s Normal ScalingReplicaSet deployment/nfd-gc Scaled up replica set nfd-gc-7d6955597d from 0 to 1 +node-feature-discovery 4m48s Normal SuccessfulCreate replicaset/nfd-gc-7d6955597d Created pod: nfd-gc-7d6955597d-v5ns2 +kube-system 4m48s Normal SuccessfulCreate job/helm-install-gateway-api-crd Created pod: helm-install-gateway-api-crd-gfhvl +kube-system 4m48s Normal Resumed job/helm-install-gateway-api-crd Job resumed +kube-system 4m48s Normal ScalingReplicaSet deployment/local-path-provisioner Scaled up replica set local-path-provisioner-6858d854cf from 0 to 1 +kube-system 4m48s Normal ScalingReplicaSet deployment/coredns Scaled up replica set coredns-577d995dff from 0 to 1 +kube-system 4m48s Normal AppliedManifest addon/runtimes Applied manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml" +kube-system 4m48s Normal ScalingReplicaSet deployment/metrics-server Scaled up replica set metrics-server-778c4649b4 from 0 to 1 +kube-system 4m48s Normal ApplyingManifest addon/runtimes Applying manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml" +kube-system 4m48s Warning FailedCreatePodSandBox pod/intel-gpu-plugin-hxvpb Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0b8e1ef825f5cf964109182847df4ec64752f3a83a9d675d98dd47e1e78e567b": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m48s Normal AppliedManifest addon/rolebindings Applied manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml" +kube-system 4m48s Normal ApplyingManifest addon/rolebindings Applying manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml" +kube-system 4m48s Normal AppliedManifest addon/nvidia-device-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml" +kube-system 4m48s Normal ApplyingManifest addon/nvidia-device-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml" +kube-system 4m48s Normal DeletingManifest addon/nri-balloons Deleting manifest at "/var/lib/rancher/k3s/server/manifests/nri-balloons.yaml" +kube-system 4m48s Normal AppliedManifest addon/node-feature-discovery Applied manifest at "/var/lib/rancher/k3s/server/manifests/node-feature-discovery.yaml" +kube-system 4m48s Normal SuccessfulCreate daemonset/intel-gpu-plugin Created pod: intel-gpu-plugin-hxvpb +kube-system 4m47s Warning FailedCreatePodSandBox pod/coredns-577d995dff-c89zz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ee35c4604fe63c0786db2ce1fa271557f8e4094f1d63a6b3b18a7ee8591d15a5": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m47s Warning FailedCreatePodSandBox pod/local-path-provisioner-6858d854cf-8t662 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b404afdd943e2e02af64bff258bd72ff4900fdfe152117f2e4d2d8fb3c9a4fad": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m47s Warning FailedCreatePodSandBox pod/metrics-server-778c4649b4-l4d4j Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e8d628ccf8e639859e8df7149b7be5f14fc617e75945c4d6406e2ab6c659505b": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 4m47s Warning FailedCreatePodSandBox pod/nfd-gc-7d6955597d-v5ns2 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8d497575b988adc57da6778427415eae95251b88ef75ff3d315f14ad3faf4ddc": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 4m47s Warning FailedCreatePodSandBox pod/nfd-master-74b6cfb7df-m4whr Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1ffbc73bc8e37dd138d5b283910fd2d5b7a176dd1dba68a00c2d024e22974cb2": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m34s Normal Pulling pod/metrics-server-778c4649b4-l4d4j Pulling image "rancher/mirrored-metrics-server:v0.9.0" +kube-system 4m34s Normal Pulling pod/intel-gpu-plugin-hxvpb Pulling image "intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33" +kube-system 4m34s Normal Pulling pod/local-path-provisioner-6858d854cf-8t662 Pulling image "rancher/local-path-provisioner:v0.0.37" +kube-system 4m34s Normal Pulling pod/coredns-577d995dff-c89zz Pulling image "rancher/mirrored-coredns-coredns:1.14.7" +node-feature-discovery 4m34s Normal Pulling pod/nfd-gc-7d6955597d-v5ns2 Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +node-feature-discovery 4m34s Normal Pulling pod/nfd-worker-87d64 Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +kube-system 4m34s Normal Pulling pod/helm-install-gateway-api-crd-gfhvl Pulling image "rancher/klipper-helm:v0.13.3-build20260727" +node-feature-discovery 4m32s Normal Pulling pod/nfd-master-74b6cfb7df-m4whr Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +kube-system 4m31s Normal Started pod/intel-gpu-plugin-hxvpb Container started +kube-system 4m31s Normal Created pod/intel-gpu-plugin-hxvpb Container created +kube-system 4m31s Normal Pulled pod/intel-gpu-plugin-hxvpb Successfully pulled image "intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33" in 2.652s (2.652s including waiting). Image size: 5970136 bytes. +node-feature-discovery 4m30s Normal Started pod/nfd-master-74b6cfb7df-m4whr Container started +node-feature-discovery 4m30s Normal Created pod/nfd-master-74b6cfb7df-m4whr Container created +node-feature-discovery 4m30s Normal Pulled pod/nfd-master-74b6cfb7df-m4whr Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 1.71s (1.71s including waiting). Image size: 85412382 bytes. +node-feature-discovery 4m30s Warning Unhealthy pod/nfd-master-74b6cfb7df-m4whr Readiness probe failed: Get "http://10.42.0.9:8080/healthz": dial tcp 10.42.0.9:8080: connect: connection refused +node-feature-discovery 4m30s Normal Started pod/nfd-gc-7d6955597d-v5ns2 Container started +node-feature-discovery 4m30s Normal Created pod/nfd-gc-7d6955597d-v5ns2 Container created +node-feature-discovery 4m30s Normal Pulled pod/nfd-gc-7d6955597d-v5ns2 Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 3.227s (3.227s including waiting). Image size: 85412382 bytes. +kube-system 4m30s Normal Created pod/local-path-provisioner-6858d854cf-8t662 Container created +kube-system 4m30s Normal Pulled pod/local-path-provisioner-6858d854cf-8t662 Successfully pulled image "rancher/local-path-provisioner:v0.0.37" in 4.181s (4.181s including waiting). Image size: 15066012 bytes. +node-feature-discovery 4m30s Normal Pulled pod/nfd-worker-87d64 Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 3.112s (3.112s including waiting). Image size: 85412382 bytes. +node-feature-discovery 4m30s Normal Created pod/nfd-worker-87d64 Container created +node-feature-discovery 4m30s Normal Started pod/nfd-worker-87d64 Container started +kube-system 4m29s Normal Created pod/helm-install-gateway-api-crd-gfhvl Container created +kube-system 4m29s Warning Unhealthy pod/metrics-server-778c4649b4-l4d4j Readiness probe failed: Get "https://10.42.0.3:10250/readyz": dial tcp 10.42.0.3:10250: connect: connection refused +kube-system 4m29s Normal Started pod/helm-install-gateway-api-crd-gfhvl Container started +kube-system 4m29s Normal Created pod/coredns-577d995dff-c89zz Container created +kube-system 4m29s Normal Pulled pod/helm-install-gateway-api-crd-gfhvl Successfully pulled image "rancher/klipper-helm:v0.13.3-build20260727" in 4.831s (4.831s including waiting). Image size: 74172006 bytes. +kube-system 4m29s Warning Unhealthy pod/coredns-577d995dff-c89zz Readiness probe failed: Get "http://10.42.0.2:8181/ready": dial tcp 10.42.0.2:8181: connect: connection refused +kube-system 4m29s Normal Pulled pod/metrics-server-778c4649b4-l4d4j Successfully pulled image "rancher/mirrored-metrics-server:v0.9.0" in 4.275s (4.275s including waiting). Image size: 23258428 bytes. +kube-system 4m29s Normal Pulled pod/coredns-577d995dff-c89zz Successfully pulled image "rancher/mirrored-coredns-coredns:1.14.7" in 4.936s (4.936s including waiting). Image size: 24261571 bytes. +kube-system 4m29s Normal Created pod/metrics-server-778c4649b4-l4d4j Container created +kube-system 4m29s Normal Started pod/local-path-provisioner-6858d854cf-8t662 Container started +kube-system 4m29s Normal Started pod/coredns-577d995dff-c89zz Container started +kube-system 4m29s Normal Started pod/metrics-server-778c4649b4-l4d4j Container started +kube-system 4m26s Normal Completed job/helm-install-gateway-api-crd Job completed +default 4m21s Normal Pulling pod/gpu-hold-4 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 4m21s Normal Pulling pod/gpu-hold-2 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 4m21s Normal Pulling pod/gpu-hold-1 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 4m21s Normal Pulling pod/gpu-hold-3 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 4m19s Normal Pulled pod/gpu-hold-4 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.802s (1.802s including waiting). Image size: 2222118 bytes. +default 4m19s Normal Created pod/gpu-hold-1 Container created +default 4m19s Normal Created pod/gpu-hold-4 Container created +default 4m19s Normal Started pod/gpu-hold-1 Container started +default 4m19s Normal Pulled pod/gpu-hold-2 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.941s (1.941s including waiting). Image size: 2222118 bytes. +default 4m19s Normal Started pod/gpu-hold-3 Container started +default 4m19s Normal Pulled pod/gpu-hold-3 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.88s (1.88s including waiting). Image size: 2222118 bytes. +default 4m19s Normal Created pod/gpu-hold-2 Container created +default 4m19s Normal Started pod/gpu-hold-2 Container started +default 4m19s Normal Pulled pod/gpu-hold-1 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 2.004s (2.004s including waiting). Image size: 2222118 bytes. +default 4m19s Normal Started pod/gpu-hold-4 Container started +default 4m19s Normal Created pod/gpu-hold-3 Container created +default 4m17s Normal Killing pod/gpu-hold-1 Stopping container hold +kube-system 4m14s Warning Unhealthy pod/metrics-server-778c4649b4-l4d4j Readiness probe failed: HTTP probe failed with statuscode: 500 +default 3m46s Normal Killing pod/gpu-hold-2 Stopping container hold +default 3m15s Normal Killing pod/gpu-hold-3 Stopping container hold +default 2m44s Normal Killing pod/gpu-hold-4 Stopping container hold +default 2m13s Normal ScalingReplicaSet deployment/vllm-qwen38-27b-xpu Scaled up replica set vllm-qwen38-27b-xpu-799dc85d7c from 0 to 1 +default 2m13s Normal SuccessfulCreate replicaset/vllm-qwen38-27b-xpu-799dc85d7c Created pod: vllm-qwen38-27b-xpu-799dc85d7c-ng77g +default 2m12s Normal Pulling pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Pulling image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" +default 28s Normal Pulled pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Successfully pulled image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" in 1m43.931s (1m43.931s including waiting). Image size: 4143776505 bytes. +default 11s Normal Created pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Container created +default 11s Normal Started pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Container started +default 11s Normal Pulled pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Container image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" already present on machine and can be accessed by the pod +default 10s Warning BackOff pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Back-off restarting failed container vllm in pod vllm-qwen38-27b-xpu-799dc85d7c-ng77g_default(6077fde9-5952-450e-b869-3aa47c254b6a) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/gpu-hold.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/gpu-hold.txt new file mode 100644 index 00000000..19ea6376 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/gpu-hold.txt @@ -0,0 +1,4 @@ +gpu-hold-1 renderD128 0000:12:00.0 +gpu-hold-2 renderD130 0000:7c:00.0 +gpu-hold-3 renderD131 0000:96:00.0 +gpu-hold-4 renderD129 0000:48:00.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/live.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/live.log new file mode 100644 index 00000000..177e6bd6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/live.log @@ -0,0 +1,5 @@ +== arm H53 start 00:20:14 (+0s) +== k3d-gpu up 00:20:15 (+1s) +== wait-ready 00:23:06 (+172s) +== cleanup 00:25:19 (+305s) +== done 00:25:22 (+308s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.previous.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.previous.txt new file mode 100644 index 00000000..597ea0e6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.previous.txt @@ -0,0 +1 @@ +--: line 1: /opt/intel/oneapi/setvars.sh: No such file or directory diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt new file mode 100644 index 00000000..597ea0e6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt @@ -0,0 +1 @@ +--: line 1: /opt/intel/oneapi/setvars.sh: No such file or directory diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pin.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pin.yaml new file mode 100644 index 00000000..96094b70 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pin.yaml @@ -0,0 +1,2 @@ +image: + tag: "v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pods.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pods.txt new file mode 100644 index 00000000..94ce1df0 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pods.txt @@ -0,0 +1,10 @@ +NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +default vllm-qwen38-27b-xpu-799dc85d7c-ng77g 0/1 CrashLoopBackOff 2 (11s ago) 2m13s 10.42.0.14 k3d-k3dgpu-e-h53-server-0 +kube-system coredns-577d995dff-c89zz 1/1 Running 0 4m48s 10.42.0.2 k3d-k3dgpu-e-h53-server-0 +kube-system helm-install-gateway-api-crd-gfhvl 0/1 Completed 0 4m48s 10.42.0.4 k3d-k3dgpu-e-h53-server-0 +kube-system intel-gpu-plugin-hxvpb 1/1 Running 0 4m48s 10.42.0.5 k3d-k3dgpu-e-h53-server-0 +kube-system local-path-provisioner-6858d854cf-8t662 1/1 Running 0 4m48s 10.42.0.6 k3d-k3dgpu-e-h53-server-0 +kube-system metrics-server-778c4649b4-l4d4j 1/1 Running 0 4m48s 10.42.0.3 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-gc-7d6955597d-v5ns2 1/1 Running 0 4m48s 10.42.0.8 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-master-74b6cfb7df-m4whr 1/1 Running 0 4m48s 10.42.0.9 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-worker-87d64 1/1 Running 0 4m48s 10.42.0.7 k3d-k3dgpu-e-h53-server-0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/reset-mcp.sh b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/reset-mcp.sh new file mode 100755 index 00000000..9730153a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/reset-mcp.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +kill "$(cat "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold/mcp.pid")" 2>/dev/null +for _ in $(seq 50); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null || break; sleep 0.1; done +cd "/home/devcloud/k3dgpu-bench/eab/usecases/banking-assistant/src" && (MCP_HOST=127.0.0.1 MCP_PORT=18600 exec "/home/devcloud/k3dgpu-bench/mafvenv/bin/python" -m mcp_server.server >> "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold/mcp.log" 2>&1 & + echo $! > "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold/mcp.pid") +for _ in $(seq 120); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null && exit 0; sleep 0.5; done +exit 1 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/up.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/up.log new file mode 100644 index 00000000..06d0310e --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/up.log @@ -0,0 +1,32 @@ +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators: intel amx +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-e-h53' with image cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-e-h53' +INFO[0000] Created image volume k3d-k3dgpu-e-h53-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-e-h53-tools' +INFO[0001] Creating node 'k3d-k3dgpu-e-h53-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-e-h53' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-e-h53-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-e-h53' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-e-h53) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold/kubeconfig +==> Waiting for nodes Ready... +node/k3d-k3dgpu-e-h53-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +==> intel device plugin auto-deployed by k3s. +==> Verifying accelerators are schedulable... +daemon set "intel-gpu-plugin" successfully rolled out +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators schedulable: intel=4 amx=1. Try: k3d-gpu test diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/b-listing.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/b-listing.txt new file mode 100644 index 00000000..de902c4d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/b-listing.txt @@ -0,0 +1,34 @@ +.gitattributes 1570 1793310278.7959648510 +LICENSE 11544 1793310278.8089651410 +README.md 65012 1793310278.8199653860 +chat_template.jinja 8952 1793310278.8139652520 +config.json 4312 1793310278.8129652300 +crc32.txt 238 1793310278.8299656090 +generation_config.json 202 1793310278.8259655200 +merges.txt 3353259 1793310278.8589662550 +model-00001-of-00018.safetensors 3966730552 1793310507.8016561670 +model-00002-of-00018.safetensors 3043080328 1793310511.8507358520 +model-00003-of-00018.safetensors 2542796952 1793310459.7537105830 +model-00004-of-00018.safetensors 3988973152 1793310571.2509048500 +model-00005-of-00018.safetensors 2099339864 1793310430.8621419970 +model-00006-of-00018.safetensors 3979553696 1793310509.6426923980 +model-00007-of-00018.safetensors 2108759344 1793310436.7122571270 +model-00008-of-00018.safetensors 3979553696 1793310573.9919587940 +model-00009-of-00018.safetensors 2108759344 1793310591.7193076680 +model-00010-of-00018.safetensors 3979553696 1793310721.1158541950 +model-00011-of-00018.safetensors 2108759344 1793310632.3041063770 +model-00012-of-00018.safetensors 3979553696 1793310730.4580380490 +model-00013-of-00018.safetensors 2108759344 1793310660.3256578430 +model-00014-of-00018.safetensors 3979553696 1793310752.6844754650 +model-00015-of-00018.safetensors 2108759344 1793310664.6107421730 +model-00016-of-00018.safetensors 3979564040 1793310758.3635872300 +model-00017-of-00018.safetensors 2108759344 1793310689.8802394770 +model-00018-of-00018.safetensors 3392197344 1793310751.2064463780 +model.safetensors.index.json 112216 1793310660.4686606570 +preprocessor_config.json 390 1793310660.5806628610 +tokenizer.json 12809320 1793310661.7056850020 +tokenizer_config.json 17928 1793310661.8266873830 +video_preprocessor_config.json 385 1793310661.9656901190 +vocab.json 6722759 1793310662.3626979320 +77042094076611b69791a610065f28b7013b8c621795fa86ddccc8bac7d1b9df model.safetensors.index.json +191e0af232104ed8b65258cf3fb2b842e288008baca7633c11b82a1ac7203aab config.json diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/blueprint-commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/blueprint-commit.txt new file mode 100644 index 00000000..2cf3b989 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/blueprint-commit.txt @@ -0,0 +1 @@ +758d4845520ca5456bf732264ef02d92a26d25a6 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/commit.txt new file mode 100644 index 00000000..4af4a96a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/commit.txt @@ -0,0 +1 @@ +1f39488be7695dc0d1848347bf1c91f04025bc55 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/deploy.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/deploy.yaml new file mode 100644 index 00000000..864f6f94 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/deploy.yaml @@ -0,0 +1,188 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: vllm-qwen38-27b-xpu-config + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +data: + http_proxy: '' + https_proxy: '' + no_proxy: '' + TORCHINDUCTOR_CACHE_DIR: /tmp + NUMBA_CACHE_DIR: /tmp + HF_HOME: /data + OUTLINES_CACHE_DIR: /tmp/.cache/outlines + VLLM_NO_USAGE_STATS: '1' + DO_NOT_TRACK: '1' + CCL_ALLREDUCE: ring + CCL_PROCESS_LAUNCHER: none + CCL_WORKER_AFFINITY: '' + CCL_WORKER_OFFLOAD: '0' + CCL_ZE_IPC_EXCHANGE: pidfd + HF_ENDPOINT: https://huggingface.co + HF_HUB_DISABLE_XET: '1' + HF_HUB_ENABLE_HF_TRANSFER: '0' + HOME: /tmp + LOGNAME: vllm + ONEAPI_DEVICE_SELECTOR: level_zero:* + SYCL_CACHE_PERSISTENT: '1' + TORCH_LLM_ALLREDUCE: '1' + TRITON_CACHE_DIR: /tmp/.triton + TRITON_HOME: /tmp/.triton + USER: vllm + VLLM_ALLOW_LONG_MAX_MODEL_LEN: '1' + VLLM_ATTENTION_BACKEND: TORCH_SDPA + VLLM_TARGET_DEVICE: xpu + VLLM_USE_V1: '1' + VLLM_WORKER_MULTIPROC_METHOD: spawn + XDG_CACHE_HOME: /tmp/.cache + ZES_ENABLE_SYSMAN: '1' + ZE_ENABLE_PCI_ID_DEVICE_ORDER: '1' + ZE_FLAT_DEVICE_HIERARCHY: FLAT + HF_HUB_OFFLINE: '1' +--- +apiVersion: v1 +kind: Service +metadata: + name: vllm-qwen38-27b-xpu-service + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 8000 + protocol: TCP + name: vllm + selector: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-qwen38-27b-xpu + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +spec: + selector: + matchLabels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + template: + metadata: + annotations: null + labels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + spec: + securityContext: + fsGroup: 1001 + supplementalGroups: + - 990 + - 44 + containers: + - name: vllm + envFrom: + - configMapRef: + name: vllm-qwen38-27b-xpu-config + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - SYS_NICE + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: false + runAsUser: 0 + seccompProfile: + type: RuntimeDefault + image: vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + imagePullPolicy: IfNotPresent + command: + - bash + - -c + - python3 -m vllm.entrypoints.openai.api_server "$@" + - -- + args: + - --host + - 0.0.0.0 + - --dtype + - float16 + - --block-size + - '64' + - --gpu-memory-utilization + - '0.9' + - --enforce-eager + - --no-enable-prefix-caching + - --trust-remote-code + - --disable-sliding-window + - --enable-auto-tool-choice + - --tool-call-parser + - qwen3_xml + - --tensor-parallel-size + - '4' + - --model + - /data/Qwen3.8-27B + - --served-model-name + - Qwen/Qwen3.6-27B + - --port + - '8000' + volumeMounts: + - mountPath: /data + name: model-volume + - mountPath: /dev/shm + name: shm + - mountPath: /tmp + name: tmp + - mountPath: /root/.cache/vllm + name: vllm-cache + ports: + - name: http + containerPort: 8000 + protocol: TCP + readinessProbe: + failureThreshold: 200 + httpGet: + path: /health + port: http + initialDelaySeconds: 10 + periodSeconds: 5 + resources: + limits: + gpu.intel.com/xe: '4' + volumes: + - name: model-volume + hostPath: + path: /var/apb-ro + type: Directory + - name: shm + emptyDir: + medium: Memory + sizeLimit: 1Gi + - name: tmp + emptyDir: {} + - name: vllm-cache + emptyDir: {} + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: ei-inference-eligible + operator: In + values: + - 'true' diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/describe-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/describe-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt new file mode 100644 index 00000000..1743b33f --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/describe-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt @@ -0,0 +1,119 @@ +Name: vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz +Namespace: default +Priority: 0 +Service Account: default +Node: k3d-k3dgpu-e-h53-server-0/172.19.0.2 +Start Time: Fri, 30 Oct 2026 00:28:27 +0000 +Labels: app.kubernetes.io/instance=vllm-qwen38-27b-xpu + app.kubernetes.io/name=vllm + pod-template-hash=69c59cbbdc +Annotations: +Status: Running +IP: 10.42.0.14 +IPs: + IP: 10.42.0.14 +Controlled By: ReplicaSet/vllm-qwen38-27b-xpu-69c59cbbdc +Containers: + vllm: + Container ID: containerd://3eae540505cef3060d89c39dbcda364b817f1c88832c85389570dc07ce3514c5 + Image: vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + Image ID: docker.io/vllm/vllm-openai-xpu@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + Port: 8000/TCP (http) + Host Port: 0/TCP (http) + SeccompProfile: RuntimeDefault + Command: + bash + -c + python3 -m vllm.entrypoints.openai.api_server "$@" + -- + Args: + --host + 0.0.0.0 + --dtype + float16 + --block-size + 64 + --gpu-memory-utilization + 0.9 + --enforce-eager + --no-enable-prefix-caching + --trust-remote-code + --disable-sliding-window + --enable-auto-tool-choice + --tool-call-parser + qwen3_xml + --tensor-parallel-size + 4 + --model + /data/Qwen3.8-27B + --served-model-name + Qwen/Qwen3.6-27B + --port + 8000 + State: Running + Started: Fri, 30 Oct 2026 00:36:01 +0000 + Last State: Terminated + Reason: Error + Exit Code: 1 + Started: Fri, 30 Oct 2026 00:33:56 +0000 + Finished: Fri, 30 Oct 2026 00:35:32 +0000 + Ready: False + Restart Count: 3 + Limits: + gpu.intel.com/xe: 4 + Requests: + gpu.intel.com/xe: 4 + Readiness: http-get http://:http/health delay=10s timeout=1s period=5s successThreshold=1 failureThreshold=200 + Environment Variables from: + vllm-qwen38-27b-xpu-config ConfigMap Optional: false + Environment: + Mounts: + /data from model-volume (rw) + /dev/shm from shm (rw) + /root/.cache/vllm from vllm-cache (rw) + /tmp from tmp (rw) + /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hq2fx (ro) +Conditions: + Type Status + PodReadyToStartContainers True + Initialized True + Ready False + ContainersReady False + PodScheduled True +Volumes: + model-volume: + Type: HostPath (bare host directory volume) + Path: /var/apb-ro + HostPathType: Directory + shm: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: Memory + SizeLimit: 1Gi + tmp: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: + vllm-cache: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: + kube-api-access-hq2fx: + Type: Projected (a volume that contains injected data from multiple sources) + TokenExpirationSeconds: 3607 + ConfigMapName: kube-root-ca.crt + Optional: false + DownwardAPI: true +QoS Class: BestEffort +Node-Selectors: +Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s + node.kubernetes.io/unreachable:NoExecute op=Exists for 300s +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 7m37s default-scheduler Successfully assigned default/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz to k3d-k3dgpu-e-h53-server-0 + Normal Pulling 7m37s kubelet spec.containers{vllm}: Pulling image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" + Normal Pulled 5m52s kubelet spec.containers{vllm}: Successfully pulled image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" in 1m44.709s (1m44.709s including waiting). Image size: 4143776505 bytes. + Warning Unhealthy 37s (x58 over 5m41s) kubelet spec.containers{vllm}: Readiness probe failed: Get "http://10.42.0.14:8000/health": dial tcp 10.42.0.14:8000: connect: connection refused + Normal Created 3s (x4 over 5m52s) kubelet spec.containers{vllm}: Container created + Normal Started 3s (x4 over 5m52s) kubelet spec.containers{vllm}: Container started + Normal Pulled 3s (x3 over 3m57s) kubelet spec.containers{vllm}: Container image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" already present on machine and can be accessed by the pod diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/devdri-diag.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/devdri-diag.txt new file mode 100644 index 00000000..02568fcc --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/devdri-diag.txt @@ -0,0 +1,72 @@ +2026-10-30T00:36:13Z +## host: ls -la /dev/dri /dev/dri/by-path +/dev/dri: +total 0 +drwxr-xr-x 3 root root 240 Sep 15 22:48 . +drwxr-xr-x 19 root root 6480 Oct 29 18:47 .. +drwxr-xr-x 2 root root 240 Sep 15 22:48 by-path +crw-rw----+ 1 root video 226, 0 Sep 16 00:04 card0 +crw-rw----+ 1 root video 226, 1 Sep 16 00:04 card1 +crw-rw----+ 1 root video 226, 2 Sep 16 00:04 card2 +crw-rw----+ 1 root video 226, 3 Sep 16 00:04 card3 +crw-rw----+ 1 root video 226, 4 Sep 16 00:04 card4 +crw-rw----+ 1 root render 226, 128 Sep 16 00:04 renderD128 +crw-rw----+ 1 root render 226, 129 Sep 16 00:04 renderD129 +crw-rw----+ 1 root render 226, 130 Sep 16 00:04 renderD130 +crw-rw----+ 1 root render 226, 131 Sep 16 00:04 renderD131 + +/dev/dri/by-path: +total 0 +drwxr-xr-x 2 root root 240 Sep 15 22:48 . +drwxr-xr-x 3 root root 240 Sep 15 22:48 .. +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:12:00.0-card -> ../card1 +lrwxrwxrwx+1 root root 13 Sep 15 22:48 pci-0000:12:00.0-render -> ../renderD128 +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:2d:00.0-card -> ../card0 +lrwxrwxrwx+1 root root 8 Sep 15 22:47 pci-0000:2d:00.0-platform-simple-framebuffer.0-card -> ../card0 +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:48:00.0-card -> ../card2 +lrwxrwxrwx+1 root root 13 Sep 15 22:48 pci-0000:48:00.0-render -> ../renderD129 +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:7c:00.0-card -> ../card3 +lrwxrwxrwx+1 root root 13 Sep 15 22:48 pci-0000:7c:00.0-render -> ../renderD130 +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:96:00.0-card -> ../card4 +lrwxrwxrwx+1 root root 13 Sep 15 22:48 pci-0000:96:00.0-render -> ../renderD131 +## node k3d-k3dgpu-e-h53-server-0: ls -la /dev/dri /dev/dri/by-path +/dev/dri: +total 0 +drwxr-xr-x 2 root root 220 Oct 30 00:25 . +drwxr-xr-x 14 root root 6300 Oct 30 00:25 .. +crw-rw---- 1 root video 226, 0 Oct 30 00:25 card0 +crw-rw---- 1 root video 226, 1 Oct 30 00:25 card1 +crw-rw---- 1 root video 226, 2 Oct 30 00:25 card2 +crw-rw---- 1 root video 226, 3 Oct 30 00:25 card3 +crw-rw---- 1 root video 226, 4 Oct 30 00:25 card4 +crw-rw---- 1 root 990 226, 128 Oct 30 00:25 renderD128 +crw-rw---- 1 root 990 226, 129 Oct 30 00:25 renderD129 +crw-rw---- 1 root 990 226, 130 Oct 30 00:25 renderD130 +crw-rw---- 1 root 990 226, 131 Oct 30 00:25 renderD131 +ls: cannot access '/dev/dri/by-path': No such file or directory +## pod pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz: ls -la /dev/dri /dev/dri/by-path +ls: cannot access '/dev/dri/by-path': No such file or directory +/dev/dri: +total 0 +drwxr-xr-x 2 root root 200 Oct 30 00:36 . +drwxr-xr-x 6 root root 380 Oct 30 00:36 .. +crw-rw---- 1 root root 226, 1 Oct 30 00:36 card1 +crw-rw---- 1 root root 226, 2 Oct 30 00:36 card2 +crw-rw---- 1 root root 226, 3 Oct 30 00:36 card3 +crw-rw---- 1 root root 226, 4 Oct 30 00:36 card4 +crw-rw---- 1 root root 226, 128 Oct 30 00:36 renderD128 +crw-rw---- 1 root root 226, 129 Oct 30 00:36 renderD129 +crw-rw---- 1 root root 226, 130 Oct 30 00:36 renderD130 +crw-rw---- 1 root root 226, 131 Oct 30 00:36 renderD131 +command terminated with exit code 2 +## pod: /dev/dri strings in the image oneCCL libraries +/opt/venv/lib/libccl.so /opt/venv/lib/libccl.so + 1 /dev/dri/by-path/ +/opt/venv/lib/libccl.so.1 /opt/venv/lib/libccl.so.1 + 1 /dev/dri/by-path/ +/opt/venv/lib/libccl.so.1.0 /opt/venv/lib/libccl.so.1.0 + 1 /dev/dri/by-path/ +/opt/venv/lib/libccl.so.2 /opt/venv/lib/libccl.so.2 +/opt/venv/lib/libccl.so.2.0 /opt/venv/lib/libccl.so.2.0 +## intel-gpu-plugin daemonset args and volumes +intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33 [{"hostPath":{"path":"/dev/dri","type":"DirectoryOrCreate"},"name":"devfs"},{"hostPath":{"path":"/sys/class/drm","type":""},"name":"sysfsdrm"},{"hostPath":{"path":"/var/lib/kubelet/device-plugins","type":""},"name":"kubeletsockets"},{"hostPath":{"path":"/var/run/cdi","type":"DirectoryOrCreate"},"name":"cdipath"}] diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/eat.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/eat.yaml new file mode 100644 index 00000000..7c2d0407 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/eat.yaml @@ -0,0 +1,301 @@ +--- +# Source: vllm/templates/secret.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: Secret +metadata: + name: vllm-qwen38-27b-xpu-secret + labels: + app: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +type: Opaque +data: +--- +# Source: vllm/templates/configmap.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: ConfigMap +metadata: + name: vllm-qwen38-27b-xpu-config + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +data: + http_proxy: "" + https_proxy: "" + no_proxy: "" + TORCHINDUCTOR_CACHE_DIR: "/tmp" + NUMBA_CACHE_DIR: "/tmp" + HF_HOME: "/data" + OUTLINES_CACHE_DIR: "/tmp/.cache/outlines" + VLLM_NO_USAGE_STATS: "1" + DO_NOT_TRACK: "1" + CCL_ALLREDUCE: "ring" + CCL_PROCESS_LAUNCHER: "none" + CCL_WORKER_AFFINITY: "" + CCL_WORKER_OFFLOAD: "0" + CCL_ZE_IPC_EXCHANGE: "pidfd" + DO_NOT_TRACK: "1" + HF_ENDPOINT: "https://huggingface.co" + HF_HUB_DISABLE_XET: "1" + HF_HUB_ENABLE_HF_TRANSFER: "0" + HOME: "/tmp" + LOGNAME: "vllm" + ONEAPI_DEVICE_SELECTOR: "level_zero:*" + SYCL_CACHE_PERSISTENT: "1" + TORCH_LLM_ALLREDUCE: "1" + TRITON_CACHE_DIR: "/tmp/.triton" + TRITON_HOME: "/tmp/.triton" + USER: "vllm" + VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1" + VLLM_ATTENTION_BACKEND: "TORCH_SDPA" + VLLM_NO_USAGE_STATS: "1" + VLLM_TARGET_DEVICE: "xpu" + VLLM_USE_V1: "1" + VLLM_WORKER_MULTIPROC_METHOD: "spawn" + XDG_CACHE_HOME: "/tmp/.cache" + ZES_ENABLE_SYSMAN: "1" + ZE_ENABLE_PCI_ID_DEVICE_ORDER: "1" + ZE_FLAT_DEVICE_HIERARCHY: "FLAT" +--- +# Source: vllm/templates/pvc.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: vllm-qwen38-27b-xpu-pvc + namespace: default +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 250Gi # Adjust this according to model storage size +--- +# Source: vllm/templates/service.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: v1 +kind: Service +metadata: + name: vllm-qwen38-27b-xpu-service + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 8000 + protocol: TCP + name: vllm + selector: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu +--- +# Source: vllm/templates/deployment.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-qwen38-27b-xpu + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +spec: + selector: + matchLabels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + template: + metadata: + annotations: + labels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + spec: + # + # serviceAccountName: default + securityContext: + fsGroup: 1001 + supplementalGroups: + - 990 + - 44 + containers: + - name: vllm + envFrom: + - configMapRef: + name: vllm-qwen38-27b-xpu-config + - secretRef: + name: vllm-qwen38-27b-xpu-secret + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - SYS_NICE + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: false + runAsUser: 0 + seccompProfile: + type: RuntimeDefault + image: "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" + imagePullPolicy: IfNotPresent + command: + - bash + - -c + - source /opt/intel/oneapi/setvars.sh --force && export LD_LIBRARY_PATH=/host-gpu-libs:${LD_LIBRARY_PATH} + && python3 -m vllm.entrypoints.openai.api_server "$@" + - -- + # command: + # - /bin/bash + # - -c + # - | + # python3 -m vllm.entrypoints.openai.api_server --dtype float16 --model Qwen/Qwen3.6-27B --port 8000 --tensor-parallel-size 4 --block-size 64 --max-model-len 32768 --disable-log-requests + args: + - "--host" + - "0.0.0.0" + - "--dtype" + - "float16" + - "--block-size" + - "64" + - "--gpu-memory-utilization" + - "0.9" + - "--enforce-eager" + - "--no-enable-prefix-caching" + - "--trust-remote-code" + - "--disable-sliding-window" + - "--enable-auto-tool-choice" + - "--tool-call-parser" + - "qwen3_xml" + - "--tensor-parallel-size" + - "4" + - "--model" + - "Qwen/Qwen3.6-27B" + - --served-model-name + - "Qwen/Qwen3.6-27B" + - "--port" + - "8000" + volumeMounts: + - mountPath: /data + name: model-volume + - mountPath: /dev/shm + name: shm + - mountPath: /tmp + name: tmp + - mountPath: /root/.cache/vllm + name: vllm-cache + - mountPath: /host-gpu-libs + name: host-gpu-libs + readOnly: true + - mountPath: /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1 + name: host-gpu-libs + readOnly: true + subPath: libze_intel_gpu.so.1 + - mountPath: /usr/lib/x86_64-linux-gnu/libze_loader.so.1 + name: host-gpu-libs + readOnly: true + subPath: libze_loader.so.1 + - mountPath: /etc/OpenCL/vendors + name: host-opencl-vendors + readOnly: true + - mountPath: /usr/lib/x86_64-linux-gnu/intel-opencl + name: host-intel-opencl + readOnly: true + ports: + - name: http + containerPort: 8000 + protocol: TCP + readinessProbe: + failureThreshold: 98 + httpGet: + path: /health + port: http + initialDelaySeconds: 120 + periodSeconds: 30 + resources: + limits: + "gpu.intel.com/xe": "4" + + volumes: + - name: model-volume + persistentVolumeClaim: + claimName: vllm-qwen38-27b-xpu-pvc + - name: shm + emptyDir: + medium: Memory + sizeLimit: 1Gi + - name: tmp + emptyDir: {} + - name: vllm-cache + emptyDir: {} + - hostPath: + path: /usr/lib/x86_64-linux-gnu + type: Directory + name: host-gpu-libs + - hostPath: + path: /etc/OpenCL/vendors + type: Directory + name: host-opencl-vendors + - hostPath: + path: /usr/lib/x86_64-linux-gnu/intel-opencl + type: Directory + name: host-intel-opencl + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: ei-inference-eligible + operator: In + values: + - "true" + +--- +# Source: vllm/templates/apisixroutes.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +--- +# Source: vllm/templates/chat-template-configmap.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +--- +# Source: vllm/templates/ingress.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# Please edit the object below. Lines beginning with a '#' will be ignored, +# and an empty file will abort the edit. If an error occurs while saving this file will be +# reopened with the relevant failures. +--- +# Source: vllm/templates/ingress_eks.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +--- +# Source: vllm/templates/route.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +--- +# Source: vllm/templates/servicemonitor.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/events.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/events.txt new file mode 100644 index 00000000..a98c3218 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/events.txt @@ -0,0 +1,164 @@ +NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE +node-feature-discovery 10m Normal Scheduled pod/nfd-master-74b6cfb7df-6wq4q Successfully assigned node-feature-discovery/nfd-master-74b6cfb7df-6wq4q to k3d-k3dgpu-e-h53-server-0 +default 10m Normal Starting node/k3d-k3dgpu-e-h53-server-0 +default 9m45s Normal Scheduled pod/gpu-hold-1 Successfully assigned default/gpu-hold-1 to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/intel-gpu-plugin-sfv7x Successfully assigned kube-system/intel-gpu-plugin-sfv7x to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 10m Normal Scheduled pod/nfd-worker-j8s9j Successfully assigned node-feature-discovery/nfd-worker-j8s9j to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/metrics-server-778c4649b4-gp9l4 Successfully assigned kube-system/metrics-server-778c4649b4-gp9l4 to k3d-k3dgpu-e-h53-server-0 +default 9m45s Normal Scheduled pod/gpu-hold-2 Successfully assigned default/gpu-hold-2 to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/helm-install-gateway-api-crd-n4z4l Successfully assigned kube-system/helm-install-gateway-api-crd-n4z4l to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/coredns-577d995dff-bsdh4 Successfully assigned kube-system/coredns-577d995dff-bsdh4 to k3d-k3dgpu-e-h53-server-0 +default 9m45s Normal Scheduled pod/gpu-hold-3 Successfully assigned default/gpu-hold-3 to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/local-path-provisioner-6858d854cf-wx2r9 Successfully assigned kube-system/local-path-provisioner-6858d854cf-wx2r9 to k3d-k3dgpu-e-h53-server-0 +default 9m45s Normal Scheduled pod/gpu-hold-4 Successfully assigned default/gpu-hold-4 to k3d-k3dgpu-e-h53-server-0 +default 7m37s Normal Scheduled pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Successfully assigned default/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 10m Normal Scheduled pod/nfd-gc-7d6955597d-28trj Successfully assigned node-feature-discovery/nfd-gc-7d6955597d-28trj to k3d-k3dgpu-e-h53-server-0 +default 10m Normal NodeAllocatableEnforced node/k3d-k3dgpu-e-h53-server-0 Updated Node Allocatable limit across pods +default 10m Normal NodeHasSufficientPID node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasSufficientPID +default 10m Normal NodeHasNoDiskPressure node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasNoDiskPressure +default 10m Normal CertificateExpirationOK node/k3d-k3dgpu-e-h53-server-0 Node and Certificate Authority certificates managed by k3s are OK +default 10m Normal NodeHasSufficientMemory node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasSufficientMemory +default 10m Warning InvalidDiskCapacity node/k3d-k3dgpu-e-h53-server-0 invalid capacity 0 on image filesystem +default 10m Normal Starting node/k3d-k3dgpu-e-h53-server-0 Starting kubelet. +default 10m Normal NodeReady node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeReady +kube-system 10m Normal DeletingManifest addon/intel-gpu-dra Deleting manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-dra.yaml" +kube-system 10m Normal ApplyingManifest addon/ccm Applying manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-gpu-wsl-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-wsl-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/intel-gpu-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-gpu-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/intel-dsa-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-dsa-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-dsa-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-dsa-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/gaudi-device-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/gaudi-device-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/gaudi-device-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/gaudi-device-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/gateway-api-crd Applied manifest at "/var/lib/rancher/k3s/server/manifests/gateway-api-crd.yaml" +kube-system 10m Normal ApplyingManifest addon/gateway-api-crd Applying manifest at "/var/lib/rancher/k3s/server/manifests/gateway-api-crd.yaml" +kube-system 10m Normal AppliedManifest addon/coredns Applied manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml" +kube-system 10m Normal ApplyingManifest addon/coredns Applying manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml" +kube-system 10m Normal AppliedManifest addon/ccm Applied manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml" +kube-system 10m Normal ApplyJob helmchart/gateway-api-crd Applying HelmChart from https://%{KUBERNETES_API}%/static/charts/gateway-api-crd-1.6.103.tgz using Job kube-system/helm-install-gateway-api-crd +kube-system 10m Normal ApplyingManifest addon/intel-iaa-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-iaa-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/metrics-apiservice Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" +kube-system 10m Normal ApplyingManifest addon/metrics-apiservice Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" +kube-system 10m Normal AppliedManifest addon/local-storage Applied manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml" +kube-system 10m Normal ApplyingManifest addon/local-storage Applying manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml" +kube-system 10m Normal ApplyingManifest addon/node-feature-discovery Applying manifest at "/var/lib/rancher/k3s/server/manifests/node-feature-discovery.yaml" +kube-system 10m Normal AppliedManifest addon/intel-qat-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-qat-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/metrics-server-service Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" +kube-system 10m Normal ApplyingManifest addon/aggregated-metrics-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" +kube-system 10m Normal AppliedManifest addon/aggregated-metrics-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" +kube-system 10m Normal ApplyingManifest addon/auth-delegator Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" +kube-system 10m Normal AppliedManifest addon/auth-delegator Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" +kube-system 10m Normal ApplyingManifest addon/auth-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" +kube-system 10m Normal AppliedManifest addon/auth-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-qat-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-qat-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/intel-npu-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-npu-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-npu-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-npu-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/intel-iaa-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-iaa-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/resource-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" +kube-system 10m Normal AppliedManifest addon/intel-gpu-wsl-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-wsl-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/metrics-server-deployment Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" +kube-system 10m Normal AppliedManifest addon/metrics-server-deployment Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" +kube-system 10m Normal ApplyingManifest addon/metrics-server-service Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" +kube-system 10m Normal AppliedManifest addon/resource-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" +default 10m Normal NodePasswordValidationComplete node/k3d-k3dgpu-e-h53-server-0 Deferred node password secret validation complete +default 10m Normal Synced node/k3d-k3dgpu-e-h53-server-0 Node synced successfully +kube-system 10m Normal ApplyingManifest addon/runtimes Applying manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml" +kube-system 10m Normal AppliedManifest addon/runtimes Applied manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml" +default 10m Normal RegisteredNode node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 event: Registered Node k3d-k3dgpu-e-h53-server-0 in Controller +kube-system 10m Normal ApplyingManifest addon/rolebindings Applying manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml" +kube-system 10m Normal ResumeJob helmchart/gateway-api-crd Resumed synced Job kube-system/helm-install-gateway-api-crd +kube-system 10m Normal AppliedManifest addon/nvidia-device-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/rolebindings Applied manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml" +node-feature-discovery 10m Normal SuccessfulCreate daemonset/nfd-worker Created pod: nfd-worker-j8s9j +kube-system 10m Normal DeletingManifest addon/traefik Deleting manifest at "/var/lib/rancher/k3s/server/manifests/traefik.yaml" +kube-system 10m Normal Resumed job/helm-install-gateway-api-crd Job resumed +kube-system 10m Normal ApplyingManifest addon/nvidia-device-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml" +kube-system 10m Normal DeletingManifest addon/nri-balloons Deleting manifest at "/var/lib/rancher/k3s/server/manifests/nri-balloons.yaml" +kube-system 10m Normal AppliedManifest addon/node-feature-discovery Applied manifest at "/var/lib/rancher/k3s/server/manifests/node-feature-discovery.yaml" +kube-system 10m Normal SuccessfulCreate job/helm-install-gateway-api-crd Created pod: helm-install-gateway-api-crd-n4z4l +kube-system 10m Normal Suspended job/helm-install-gateway-api-crd Job suspended +kube-system 10m Warning FailedCreatePodSandBox pod/metrics-server-778c4649b4-gp9l4 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4f8399710121c9a074b4cd33924be84f0033a52cade1a44b25cd75f00bcc4e26": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 10m Normal SuccessfulCreate replicaset/local-path-provisioner-6858d854cf Created pod: local-path-provisioner-6858d854cf-wx2r9 +kube-system 10m Normal SuccessfulCreate replicaset/metrics-server-778c4649b4 Created pod: metrics-server-778c4649b4-gp9l4 +node-feature-discovery 10m Normal ScalingReplicaSet deployment/nfd-gc Scaled up replica set nfd-gc-7d6955597d from 0 to 1 +kube-system 10m Warning FailedMount pod/helm-install-gateway-api-crd-n4z4l MountVolume.SetUp failed for volume "kube-api-access-v8jfs" : configmap "kube-root-ca.crt" not found +kube-system 10m Normal ScalingReplicaSet deployment/coredns Scaled up replica set coredns-577d995dff from 0 to 1 +node-feature-discovery 10m Warning FailedCreatePodSandBox pod/nfd-worker-j8s9j Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d0eac7f436aea07971f0b66a8a1c21ad1d05b90b6c25d391bf2847d19714265f": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 10m Normal SuccessfulCreate replicaset/nfd-master-74b6cfb7df Created pod: nfd-master-74b6cfb7df-6wq4q +kube-system 10m Normal SuccessfulCreate replicaset/coredns-577d995dff Created pod: coredns-577d995dff-bsdh4 +kube-system 10m Normal ScalingReplicaSet deployment/local-path-provisioner Scaled up replica set local-path-provisioner-6858d854cf from 0 to 1 +node-feature-discovery 10m Normal ScalingReplicaSet deployment/nfd-master Scaled up replica set nfd-master-74b6cfb7df from 0 to 1 +kube-system 10m Normal ScalingReplicaSet deployment/metrics-server Scaled up replica set metrics-server-778c4649b4 from 0 to 1 +kube-system 10m Warning FailedCreatePodSandBox pod/intel-gpu-plugin-sfv7x Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2285fe50dcd6ea62c3c74cdf96ce93d647de8dd6c94c5c46e141c8e6fb19a3fb": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 10m Warning FailedCreatePodSandBox pod/nfd-gc-7d6955597d-28trj Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5cbcac78bd49b0c5bc76d3f27c874ec598a7492981909c98dcaf26ab0feabe20": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 10m Normal SuccessfulCreate daemonset/intel-gpu-plugin Created pod: intel-gpu-plugin-sfv7x +node-feature-discovery 10m Warning FailedCreatePodSandBox pod/nfd-master-74b6cfb7df-6wq4q Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d4fde5d6750465fb7f87ea30f039d3fe65bfb08120606ad0530a0d2097fc62ad": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 10m Warning FailedCreatePodSandBox pod/local-path-provisioner-6858d854cf-wx2r9 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "705ec65010d1d8a6ffee20b2f8ea22d6931b5f638733c6a363d061b0023c0bbf": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 10m Normal SuccessfulCreate replicaset/nfd-gc-7d6955597d Created pod: nfd-gc-7d6955597d-28trj +kube-system 10m Warning FailedCreatePodSandBox pod/coredns-577d995dff-bsdh4 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "36f697d6bcaab61deb47c6276b332ff1d643e13a5da41df6027d362fe6b88f7a": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 10m Warning FailedCreatePodSandBox pod/helm-install-gateway-api-crd-n4z4l Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2cd13b2cc3035a6c20d4040c80975ad185630ea3588c7d4737923e631de4f1cb": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 9m59s Normal Pulling pod/nfd-master-74b6cfb7df-6wq4q Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +kube-system 9m59s Normal Pulling pod/coredns-577d995dff-bsdh4 Pulling image "rancher/mirrored-coredns-coredns:1.14.7" +kube-system 9m59s Normal Pulling pod/local-path-provisioner-6858d854cf-wx2r9 Pulling image "rancher/local-path-provisioner:v0.0.37" +kube-system 9m58s Normal Pulling pod/metrics-server-778c4649b4-gp9l4 Pulling image "rancher/mirrored-metrics-server:v0.9.0" +node-feature-discovery 9m58s Normal Pulling pod/nfd-worker-j8s9j Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +node-feature-discovery 9m58s Normal Pulling pod/nfd-gc-7d6955597d-28trj Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +kube-system 9m57s Normal Pulling pod/intel-gpu-plugin-sfv7x Pulling image "intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33" +kube-system 9m57s Normal Pulling pod/helm-install-gateway-api-crd-n4z4l Pulling image "rancher/klipper-helm:v0.13.3-build20260727" +kube-system 9m56s Normal Pulled pod/coredns-577d995dff-bsdh4 Successfully pulled image "rancher/mirrored-coredns-coredns:1.14.7" in 2.64s (2.64s including waiting). Image size: 24261571 bytes. +kube-system 9m56s Normal Pulled pod/local-path-provisioner-6858d854cf-wx2r9 Successfully pulled image "rancher/local-path-provisioner:v0.0.37" in 2.636s (2.636s including waiting). Image size: 15066012 bytes. +kube-system 9m56s Normal Created pod/local-path-provisioner-6858d854cf-wx2r9 Container created +kube-system 9m56s Normal Started pod/local-path-provisioner-6858d854cf-wx2r9 Container started +kube-system 9m56s Normal Created pod/coredns-577d995dff-bsdh4 Container created +kube-system 9m56s Normal Started pod/coredns-577d995dff-bsdh4 Container started +kube-system 9m56s Warning Unhealthy pod/coredns-577d995dff-bsdh4 Readiness probe failed: Get "http://10.42.0.2:8181/ready": dial tcp 10.42.0.2:8181: connect: connection refused +kube-system 9m55s Warning Unhealthy pod/metrics-server-778c4649b4-gp9l4 Readiness probe failed: Get "https://10.42.0.5:10250/readyz": dial tcp 10.42.0.5:10250: connect: connection refused +node-feature-discovery 9m55s Normal Started pod/nfd-worker-j8s9j Container started +kube-system 9m55s Normal Created pod/intel-gpu-plugin-sfv7x Container created +kube-system 9m55s Normal Pulled pod/metrics-server-778c4649b4-gp9l4 Successfully pulled image "rancher/mirrored-metrics-server:v0.9.0" in 2.962s (2.962s including waiting). Image size: 23258428 bytes. +kube-system 9m55s Normal Created pod/metrics-server-778c4649b4-gp9l4 Container created +kube-system 9m55s Normal Started pod/metrics-server-778c4649b4-gp9l4 Container started +kube-system 9m55s Normal Pulled pod/intel-gpu-plugin-sfv7x Successfully pulled image "intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33" in 2.075s (2.075s including waiting). Image size: 5970136 bytes. +node-feature-discovery 9m55s Normal Started pod/nfd-master-74b6cfb7df-6wq4q Container started +node-feature-discovery 9m55s Warning Unhealthy pod/nfd-master-74b6cfb7df-6wq4q Readiness probe failed: Get "http://10.42.0.4:8080/healthz": dial tcp 10.42.0.4:8080: connect: connection refused +node-feature-discovery 9m55s Normal Pulled pod/nfd-gc-7d6955597d-28trj Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 2.234s (2.234s including waiting). Image size: 85412382 bytes. +node-feature-discovery 9m55s Normal Created pod/nfd-gc-7d6955597d-28trj Container created +node-feature-discovery 9m55s Normal Started pod/nfd-gc-7d6955597d-28trj Container started +node-feature-discovery 9m55s Normal Pulled pod/nfd-master-74b6cfb7df-6wq4q Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 3.225s (3.225s including waiting). Image size: 85412382 bytes. +kube-system 9m55s Normal Started pod/intel-gpu-plugin-sfv7x Container started +node-feature-discovery 9m55s Normal Created pod/nfd-worker-j8s9j Container created +node-feature-discovery 9m55s Normal Pulled pod/nfd-worker-j8s9j Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 2.229s (2.229s including waiting). Image size: 85412382 bytes. +node-feature-discovery 9m55s Normal Created pod/nfd-master-74b6cfb7df-6wq4q Container created +kube-system 9m53s Normal Pulled pod/helm-install-gateway-api-crd-n4z4l Successfully pulled image "rancher/klipper-helm:v0.13.3-build20260727" in 3.609s (3.609s including waiting). Image size: 74172006 bytes. +kube-system 9m53s Normal Created pod/helm-install-gateway-api-crd-n4z4l Container created +kube-system 9m53s Normal Started pod/helm-install-gateway-api-crd-n4z4l Container started +kube-system 9m50s Normal Completed job/helm-install-gateway-api-crd Job completed +default 9m45s Normal Pulling pod/gpu-hold-1 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 9m45s Normal Pulling pod/gpu-hold-3 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 9m45s Normal Pulling pod/gpu-hold-2 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 9m45s Normal Pulling pod/gpu-hold-4 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 9m43s Normal Pulled pod/gpu-hold-3 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.345s (1.345s including waiting). Image size: 2222118 bytes. +default 9m43s Normal Started pod/gpu-hold-1 Container started +default 9m43s Normal Created pod/gpu-hold-3 Container created +default 9m43s Normal Pulled pod/gpu-hold-4 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.274s (1.274s including waiting). Image size: 2222118 bytes. +default 9m43s Normal Created pod/gpu-hold-4 Container created +default 9m43s Normal Started pod/gpu-hold-4 Container started +default 9m43s Normal Pulled pod/gpu-hold-1 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.502s (1.502s including waiting). Image size: 2222118 bytes. +default 9m43s Normal Started pod/gpu-hold-3 Container started +default 9m43s Normal Created pod/gpu-hold-1 Container created +default 9m43s Normal Started pod/gpu-hold-2 Container started +default 9m43s Normal Created pod/gpu-hold-2 Container created +default 9m43s Normal Pulled pod/gpu-hold-2 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.413s (1.413s including waiting). Image size: 2222118 bytes. +default 9m42s Normal Killing pod/gpu-hold-1 Stopping container hold +kube-system 9m41s Warning Unhealthy pod/metrics-server-778c4649b4-gp9l4 Readiness probe failed: HTTP probe failed with statuscode: 500 +default 9m10s Normal Killing pod/gpu-hold-2 Stopping container hold +default 8m39s Normal Killing pod/gpu-hold-3 Stopping container hold +default 8m8s Normal Killing pod/gpu-hold-4 Stopping container hold +default 7m37s Normal Pulling pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Pulling image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" +default 7m37s Normal SuccessfulCreate replicaset/vllm-qwen38-27b-xpu-69c59cbbdc Created pod: vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz +default 7m37s Normal ScalingReplicaSet deployment/vllm-qwen38-27b-xpu Scaled up replica set vllm-qwen38-27b-xpu-69c59cbbdc from 0 to 1 +default 5m52s Normal Pulled pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Successfully pulled image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" in 1m44.709s (1m44.709s including waiting). Image size: 4143776505 bytes. +default 37s Warning Unhealthy pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Readiness probe failed: Get "http://10.42.0.14:8000/health": dial tcp 10.42.0.14:8000: connect: connection refused +default 3s Normal Created pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Container created +default 3s Normal Started pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Container started +default 3s Normal Pulled pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Container image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" already present on machine and can be accessed by the pod diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/gpu-hold.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/gpu-hold.txt new file mode 100644 index 00000000..2c2ee671 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/gpu-hold.txt @@ -0,0 +1,4 @@ +gpu-hold-1 renderD129 0000:48:00.0 +gpu-hold-2 renderD131 0000:96:00.0 +gpu-hold-3 renderD128 0000:12:00.0 +gpu-hold-4 renderD130 0000:7c:00.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/live.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/live.log new file mode 100644 index 00000000..74383990 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/live.log @@ -0,0 +1,5 @@ +== arm H53 start 00:25:35 (+0s) +== k3d-gpu up 00:25:35 (+0s) +== wait-ready 00:28:27 (+172s) +== cleanup 00:36:21 (+646s) +== done 00:36:23 (+648s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.attempt2.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.attempt2.txt new file mode 100644 index 00000000..aaa58210 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.attempt2.txt @@ -0,0 +1,288 @@ +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] █ █ █▄ ▄█ +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.28.0 +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] █▄█▀ █ █ █ █ model /data/Qwen3.8-27B +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:272] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_xml', 'host': '0.0.0.0', 'model': '/data/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'float16', 'enforce_eager': True, 'disable_sliding_window': True, 'served_model_name': ['Qwen/Qwen3.6-27B'], 'tensor_parallel_size': 4, 'block_size': 64, 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': False} +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_HOST +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_PORT +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_ATTENTION_BACKEND +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_PORT +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_PORT_VLLM +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_PROTO +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_USE_V1 +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_ADDR +(APIServer pid=1) INFO 10-30 00:32:20 [model.py:672] Resolved architecture: Qwen3_5ForConditionalGeneration +(APIServer pid=1) WARNING 10-30 00:32:20 [model.py:2299] Casting torch.bfloat16 to torch.float16. +(APIServer pid=1) INFO 10-30 00:32:20 [model.py:1965] Using max model len 262144 +(APIServer pid=1) [ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(APIServer pid=1) [ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(APIServer pid=1) WARNING 10-30 00:32:22 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +(APIServer pid=1) WARNING 10-30 00:32:22 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +(APIServer pid=1) INFO 10-30 00:32:22 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(APIServer pid=1) INFO 10-30 00:32:22 [vllm.py:1533] Cudagraph is disabled under eager mode +(APIServer pid=1) WARNING 10-30 00:32:22 [xpu.py:302] XPU Graph is disabled by environment variable, please set VLLM_XPU_ENABLE_XPU_GRAPH=1 to enable it. +(APIServer pid=1) INFO 10-30 00:32:22 [xpu.py:381] XPU platform: set server shutdown_timeout=5. +(APIServer pid=1) INFO 10-30 00:32:22 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +(APIServer pid=1) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. +(EngineCore pid=242) INFO 10-30 00:32:42 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/data/Qwen3.8-27B', speculative_config=None, tokenizer='/data/Qwen3.8-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=xpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Qwen/Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(EngineCore pid=242) INFO 10-30 00:32:42 [multiproc_executor.py:150] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.42.0.14 (local), world_size=4, local_world_size=4 +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=496) INFO 10-30 00:32:58 [parallel_state.py:1638] world_size=4 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_39d479ea07474d849ac60b9c5c0bfe25 backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=571) INFO 10-30 00:33:11 [parallel_state.py:1638] world_size=4 rank=1 local_rank=1 distributed_init_method=file:///tmp/vllm_dist_39d479ea07474d849ac60b9c5c0bfe25 backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=652) INFO 10-30 00:33:24 [parallel_state.py:1638] world_size=4 rank=2 local_rank=2 distributed_init_method=file:///tmp/vllm_dist_39d479ea07474d849ac60b9c5c0bfe25 backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=733) INFO 10-30 00:33:37 [parallel_state.py:1638] world_size=4 rank=3 local_rank=3 distributed_init_method=file:///tmp/vllm_dist_39d479ea07474d849ac60b9c5c0bfe25 backend=xccl +(Worker pid=496) INFO 10-30 00:33:37 [parallel_state.py:1982] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +2026:10:30-00:33:37: 571 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:33:37: 571 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:33:37: 571 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:33:37: 571 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:33:37: 496 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:33:37: 496 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:33:37: 496 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:33:37: 496 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:33:37: 571 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:33:37: 733 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:33:37: 496 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:33:37: 733 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:33:37: 733 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:33:37: 733 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:33:37: 652 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:33:37: 652 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:33:37: 652 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:33:37: 652 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:33:37: 733 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:33:37: 652 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:33:38: 571:[1] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:33:38: 733:[3] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:33:38: 652:[2] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:33:38: 496:[0] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:33:38: 496:[0] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:33:38: 571:[1] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:33:38: 733:[3] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:33:38: 652:[2] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:33:38: 496:[0] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:33:38: 571:[1] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:33:38: 733:[3] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:33:38: 652:[2] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(EngineCore pid=242) INFO 10-30 00:33:38 [multiproc_executor.py:469] [shutdown] Executor: waiting for worker exit count=4 +(EngineCore pid=242) INFO 10-30 00:33:40 [multiproc_executor.py:476] [shutdown] Executor: all workers exited gracefully +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] EngineCore failed to start. +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] Traceback (most recent call last): +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 1315, in run_engine_core +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] return func(*args, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 1073, in __init__ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] super().__init__( +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 133, in __init__ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] self.model_executor = executor_class(vllm_config) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 116, in __init__ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] super().__init__(vllm_config) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] return func(*args, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/abstract.py", line 110, in __init__ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] self._init_executor() +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 211, in _init_executor +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] self.workers = WorkerProc.wait_for_ready(unready_workers) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 805, in wait_for_ready +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] raise e from None +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. +(EngineCore pid=242) Process EngineCore: +(EngineCore pid=242) Traceback (most recent call last): +(EngineCore pid=242) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap +(EngineCore pid=242) self.run() +(EngineCore pid=242) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run +(EngineCore pid=242) self._target(*self._args, **self._kwargs) +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1350, in run_engine_core +(EngineCore pid=242) raise e +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1315, in run_engine_core +(EngineCore pid=242) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) return func(*args, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1073, in __init__ +(EngineCore pid=242) super().__init__( +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 133, in __init__ +(EngineCore pid=242) self.model_executor = executor_class(vllm_config) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 116, in __init__ +(EngineCore pid=242) super().__init__(vllm_config) +(EngineCore pid=242) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) return func(*args, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/abstract.py", line 110, in __init__ +(EngineCore pid=242) self._init_executor() +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 211, in _init_executor +(EngineCore pid=242) self.workers = WorkerProc.wait_for_ready(unready_workers) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 805, in wait_for_ready +(EngineCore pid=242) raise e from None +(EngineCore pid=242) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. +(APIServer pid=1) Traceback (most recent call last): +(APIServer pid=1) File "", line 198, in _run_module_as_main +(APIServer pid=1) File "", line 88, in _run_code +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 779, in +(APIServer pid=1) uvloop.run(run_server(args)) +(APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run +(APIServer pid=1) return __asyncio.run( +(APIServer pid=1) ^^^^^^^^^^^^^^ +(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run +(APIServer pid=1) return runner.run(main) +(APIServer pid=1) ^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run +(APIServer pid=1) return self._loop.run_until_complete(task) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete +(APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper +(APIServer pid=1) return await main +(APIServer pid=1) ^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 739, in run_server +(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 753, in run_server_worker +(APIServer pid=1) async with build_async_engine_client( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=1) return await anext(self.gen) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 127, in build_async_engine_client +(APIServer pid=1) async with build_async_engine_client_from_engine_args( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=1) return await anext(self.gen) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 163, in build_async_engine_client_from_engine_args +(APIServer pid=1) async_llm = AsyncLLM.from_vllm_config( +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 220, in from_vllm_config +(APIServer pid=1) return cls( +(APIServer pid=1) ^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 149, in __init__ +(APIServer pid=1) self.engine_core = EngineCoreClient.make_async_mp_client( +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=1) return func(*args, **kwargs) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 139, in make_async_mp_client +(APIServer pid=1) return AsyncMPClient(*client_args) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=1) return func(*args, **kwargs) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 990, in __init__ +(APIServer pid=1) super().__init__( +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 609, in __init__ +(APIServer pid=1) with launch_core_engines( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ +(APIServer pid=1) next(self.gen) +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/utils.py", line 1206, in launch_core_engines +(APIServer pid=1) wait_for_engine_startup( +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/utils.py", line 1286, in wait_for_engine_startup +(APIServer pid=1) raise RuntimeError( +(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.previous.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.previous.txt new file mode 100644 index 00000000..81c267a2 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.previous.txt @@ -0,0 +1,288 @@ +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] █ █ █▄ ▄█ +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.28.0 +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] █▄█▀ █ █ █ █ model /data/Qwen3.8-27B +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:272] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_xml', 'host': '0.0.0.0', 'model': '/data/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'float16', 'enforce_eager': True, 'disable_sliding_window': True, 'served_model_name': ['Qwen/Qwen3.6-27B'], 'tensor_parallel_size': 4, 'block_size': 64, 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': False} +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_HOST +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_PORT +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_ATTENTION_BACKEND +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_PORT +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_PORT_VLLM +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_PROTO +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_USE_V1 +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_ADDR +(APIServer pid=1) INFO 10-30 00:34:10 [model.py:672] Resolved architecture: Qwen3_5ForConditionalGeneration +(APIServer pid=1) WARNING 10-30 00:34:10 [model.py:2299] Casting torch.bfloat16 to torch.float16. +(APIServer pid=1) INFO 10-30 00:34:10 [model.py:1965] Using max model len 262144 +(APIServer pid=1) [ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(APIServer pid=1) [ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(APIServer pid=1) WARNING 10-30 00:34:12 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +(APIServer pid=1) WARNING 10-30 00:34:12 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +(APIServer pid=1) INFO 10-30 00:34:12 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(APIServer pid=1) INFO 10-30 00:34:12 [vllm.py:1533] Cudagraph is disabled under eager mode +(APIServer pid=1) WARNING 10-30 00:34:12 [xpu.py:302] XPU Graph is disabled by environment variable, please set VLLM_XPU_ENABLE_XPU_GRAPH=1 to enable it. +(APIServer pid=1) INFO 10-30 00:34:12 [xpu.py:381] XPU platform: set server shutdown_timeout=5. +(APIServer pid=1) INFO 10-30 00:34:12 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +(APIServer pid=1) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. +(EngineCore pid=242) INFO 10-30 00:34:31 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/data/Qwen3.8-27B', speculative_config=None, tokenizer='/data/Qwen3.8-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=xpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Qwen/Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(EngineCore pid=242) INFO 10-30 00:34:31 [multiproc_executor.py:150] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.42.0.14 (local), world_size=4, local_world_size=4 +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=496) INFO 10-30 00:34:47 [parallel_state.py:1638] world_size=4 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_7e64a13e5b97457ab44674bb3ab7c1fb backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=571) INFO 10-30 00:35:00 [parallel_state.py:1638] world_size=4 rank=1 local_rank=1 distributed_init_method=file:///tmp/vllm_dist_7e64a13e5b97457ab44674bb3ab7c1fb backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=658) INFO 10-30 00:35:13 [parallel_state.py:1638] world_size=4 rank=2 local_rank=2 distributed_init_method=file:///tmp/vllm_dist_7e64a13e5b97457ab44674bb3ab7c1fb backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=739) INFO 10-30 00:35:26 [parallel_state.py:1638] world_size=4 rank=3 local_rank=3 distributed_init_method=file:///tmp/vllm_dist_7e64a13e5b97457ab44674bb3ab7c1fb backend=xccl +(Worker pid=496) INFO 10-30 00:35:26 [parallel_state.py:1982] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +2026:10:30-00:35:26: 739 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:35:26: 739 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:35:26: 739 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:35:26: 739 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:35:26: 496 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:35:26: 496 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:35:26: 496 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:35:26: 496 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:35:26: 658 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:35:26: 658 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:35:26: 658 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:35:26: 658 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:35:26: 739 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:35:26: 496 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:35:26: 658 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:35:26: 571 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:35:26: 571 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:35:26: 571 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:35:26: 571 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:35:26: 571 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:35:27: 496:[0] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:35:27: 658:[2] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:35:27: 571:[1] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:35:27: 739:[3] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:35:27: 496:[0] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:35:27: 571:[1] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:35:27: 739:[3] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:35:27: 658:[2] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:35:27: 496:[0] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:35:27: 571:[1] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:35:27: 739:[3] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:35:27: 658:[2] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(EngineCore pid=242) INFO 10-30 00:35:27 [multiproc_executor.py:469] [shutdown] Executor: waiting for worker exit count=4 +(EngineCore pid=242) INFO 10-30 00:35:29 [multiproc_executor.py:476] [shutdown] Executor: all workers exited gracefully +(EngineCore pid=242) Process EngineCore: +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] EngineCore failed to start. +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] Traceback (most recent call last): +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 1315, in run_engine_core +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] return func(*args, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 1073, in __init__ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] super().__init__( +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 133, in __init__ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] self.model_executor = executor_class(vllm_config) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 116, in __init__ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] super().__init__(vllm_config) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] return func(*args, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/abstract.py", line 110, in __init__ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] self._init_executor() +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 211, in _init_executor +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] self.workers = WorkerProc.wait_for_ready(unready_workers) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 805, in wait_for_ready +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] raise e from None +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. +(EngineCore pid=242) Traceback (most recent call last): +(EngineCore pid=242) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap +(EngineCore pid=242) self.run() +(EngineCore pid=242) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run +(EngineCore pid=242) self._target(*self._args, **self._kwargs) +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1350, in run_engine_core +(EngineCore pid=242) raise e +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1315, in run_engine_core +(EngineCore pid=242) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) return func(*args, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1073, in __init__ +(EngineCore pid=242) super().__init__( +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 133, in __init__ +(EngineCore pid=242) self.model_executor = executor_class(vllm_config) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 116, in __init__ +(EngineCore pid=242) super().__init__(vllm_config) +(EngineCore pid=242) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) return func(*args, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/abstract.py", line 110, in __init__ +(EngineCore pid=242) self._init_executor() +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 211, in _init_executor +(EngineCore pid=242) self.workers = WorkerProc.wait_for_ready(unready_workers) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 805, in wait_for_ready +(EngineCore pid=242) raise e from None +(EngineCore pid=242) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. +(APIServer pid=1) Traceback (most recent call last): +(APIServer pid=1) File "", line 198, in _run_module_as_main +(APIServer pid=1) File "", line 88, in _run_code +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 779, in +(APIServer pid=1) uvloop.run(run_server(args)) +(APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run +(APIServer pid=1) return __asyncio.run( +(APIServer pid=1) ^^^^^^^^^^^^^^ +(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run +(APIServer pid=1) return runner.run(main) +(APIServer pid=1) ^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run +(APIServer pid=1) return self._loop.run_until_complete(task) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete +(APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper +(APIServer pid=1) return await main +(APIServer pid=1) ^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 739, in run_server +(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 753, in run_server_worker +(APIServer pid=1) async with build_async_engine_client( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=1) return await anext(self.gen) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 127, in build_async_engine_client +(APIServer pid=1) async with build_async_engine_client_from_engine_args( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=1) return await anext(self.gen) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 163, in build_async_engine_client_from_engine_args +(APIServer pid=1) async_llm = AsyncLLM.from_vllm_config( +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 220, in from_vllm_config +(APIServer pid=1) return cls( +(APIServer pid=1) ^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 149, in __init__ +(APIServer pid=1) self.engine_core = EngineCoreClient.make_async_mp_client( +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=1) return func(*args, **kwargs) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 139, in make_async_mp_client +(APIServer pid=1) return AsyncMPClient(*client_args) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=1) return func(*args, **kwargs) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 990, in __init__ +(APIServer pid=1) super().__init__( +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 609, in __init__ +(APIServer pid=1) with launch_core_engines( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ +(APIServer pid=1) next(self.gen) +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/utils.py", line 1206, in launch_core_engines +(APIServer pid=1) wait_for_engine_startup( +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/utils.py", line 1286, in wait_for_engine_startup +(APIServer pid=1) raise RuntimeError( +(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pin.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pin.yaml new file mode 100644 index 00000000..96094b70 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pin.yaml @@ -0,0 +1,2 @@ +image: + tag: "v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pods.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pods.txt new file mode 100644 index 00000000..571eb094 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pods.txt @@ -0,0 +1,10 @@ +NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +default vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz 0/1 Running 3 (32s ago) 7m37s 10.42.0.14 k3d-k3dgpu-e-h53-server-0 +kube-system coredns-577d995dff-bsdh4 1/1 Running 0 10m 10.42.0.2 k3d-k3dgpu-e-h53-server-0 +kube-system helm-install-gateway-api-crd-n4z4l 0/1 Completed 0 10m 10.42.0.9 k3d-k3dgpu-e-h53-server-0 +kube-system intel-gpu-plugin-sfv7x 1/1 Running 0 10m 10.42.0.8 k3d-k3dgpu-e-h53-server-0 +kube-system local-path-provisioner-6858d854cf-wx2r9 1/1 Running 0 10m 10.42.0.3 k3d-k3dgpu-e-h53-server-0 +kube-system metrics-server-778c4649b4-gp9l4 1/1 Running 0 10m 10.42.0.5 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-gc-7d6955597d-28trj 1/1 Running 0 10m 10.42.0.7 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-master-74b6cfb7df-6wq4q 1/1 Running 0 10m 10.42.0.4 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-worker-j8s9j 1/1 Running 0 10m 10.42.0.6 k3d-k3dgpu-e-h53-server-0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/reset-mcp.sh b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/reset-mcp.sh new file mode 100755 index 00000000..b21a5876 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/reset-mcp.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +kill "$(cat "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold/mcp.pid")" 2>/dev/null +for _ in $(seq 50); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null || break; sleep 0.1; done +cd "/home/devcloud/k3dgpu-bench/eab/usecases/banking-assistant/src" && (MCP_HOST=127.0.0.1 MCP_PORT=18600 exec "/home/devcloud/k3dgpu-bench/mafvenv/bin/python" -m mcp_server.server >> "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold/mcp.log" 2>&1 & + echo $! > "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold/mcp.pid") +for _ in $(seq 120); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null && exit 0; sleep 0.5; done +exit 1 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/up.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/up.log new file mode 100644 index 00000000..01829362 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/up.log @@ -0,0 +1,33 @@ +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators: intel amx +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-e-h53' with image cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-e-h53' +INFO[0000] Created image volume k3d-k3dgpu-e-h53-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-e-h53-tools' +INFO[0001] Creating node 'k3d-k3dgpu-e-h53-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-e-h53' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-e-h53-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-e-h53' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-e-h53) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold/kubeconfig +==> Waiting for nodes Ready... +node/k3d-k3dgpu-e-h53-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +==> intel device plugin auto-deployed by k3s. +==> Verifying accelerators are schedulable... +Waiting for daemon set "intel-gpu-plugin" rollout to finish: 0 of 1 updated pods are available... +daemon set "intel-gpu-plugin" successfully rolled out +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators schedulable: intel=4 amx=1. Try: k3d-gpu test diff --git a/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-AG.launch.log b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-AG.launch.log new file mode 100644 index 00000000..9b222c03 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-AG.launch.log @@ -0,0 +1,11 @@ +2026-10-30T00:15:13Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:15:13Z smoke H53-smoke-AG start +== H53-smoke-AG waiting for the lock 2026-10-30T00:15:13Z +== H53-smoke-AG lock taken 2026-10-30T00:15:13Z +== H53-smoke-AG start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=AG LIVE_CAP=2370 LIVE_HOLD=1 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER= LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold 2026-10-30T00:15:14Z +/home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/crescent/hold.sh: line 151: 1549367 Killed ( while kill -0 "${cp}" 2> /dev/null; do + sleep 30; if [ "$(disk)" -gt 97 ]; then + echo "disk $(disk)% at $(date -u +%FT%TZ)" > "${R}/disk-abort"; kill -TERM "$$"; break; + fi; +done ) 7>&- 8>&- 9>&- +== H53-smoke-AG rc=0; clusters deleted: none; containers removed: none; now: 2026-10-30T00:20:05Z diff --git a/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-a.launch.log b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-a.launch.log new file mode 100644 index 00000000..64115dbd --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-a.launch.log @@ -0,0 +1,11 @@ +2026-10-30T00:20:14Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:20:14Z smoke H53-smoke-B2-a start +== H53-smoke-B2-a waiting for the lock 2026-10-30T00:20:14Z +== H53-smoke-B2-a lock taken 2026-10-30T00:20:14Z +== H53-smoke-B2-a start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=B2 LIVE_CAP=2370 LIVE_HOLD=2 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER= LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold 2026-10-30T00:20:14Z +/home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/crescent/hold.sh: line 151: 1565528 Killed ( while kill -0 "${cp}" 2> /dev/null; do + sleep 30; if [ "$(disk)" -gt 97 ]; then + echo "disk $(disk)% at $(date -u +%FT%TZ)" > "${R}/disk-abort"; kill -TERM "$$"; break; + fi; +done ) 7>&- 8>&- 9>&- +== H53-smoke-B2-a rc=143; clusters deleted: none; containers removed: none; now: 2026-10-30T00:25:26Z diff --git a/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-b.launch.log b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-b.launch.log new file mode 100644 index 00000000..2ff4c305 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-b.launch.log @@ -0,0 +1,11 @@ +2026-10-30T00:25:35Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:25:35Z smoke H53-smoke-B2-b start +== H53-smoke-B2-b waiting for the lock 2026-10-30T00:25:35Z +== H53-smoke-B2-b lock taken 2026-10-30T00:25:35Z +== H53-smoke-B2-b start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=B2 LIVE_CAP=2370 LIVE_HOLD=2 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER=b LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold 2026-10-30T00:25:35Z +/home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/crescent/hold.sh: line 151: 1579257 Killed ( while kill -0 "${cp}" 2> /dev/null; do + sleep 30; if [ "$(disk)" -gt 97 ]; then + echo "disk $(disk)% at $(date -u +%FT%TZ)" > "${R}/disk-abort"; kill -TERM "$$"; break; + fi; +done ) 7>&- 8>&- 9>&- +== H53-smoke-B2-b rc=143; clusters deleted: none; containers removed: none; now: 2026-10-30T00:36:28Z diff --git a/bench/eatflow/results/2026-10-20/H53/window/campaign-H53.log b/bench/eatflow/results/2026-10-20/H53/window/campaign-H53.log new file mode 100644 index 00000000..4fa8d93a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/campaign-H53.log @@ -0,0 +1,12 @@ +== H53-smoke-AG waiting for the lock 2026-10-30T00:15:13Z +== H53-smoke-AG lock taken 2026-10-30T00:15:13Z +== H53-smoke-AG start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=AG LIVE_CAP=2370 LIVE_HOLD=1 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER= LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold 2026-10-30T00:15:14Z +== H53-smoke-AG rc=0; clusters deleted: none; containers removed: none; now: 2026-10-30T00:20:05Z +== H53-smoke-B2-a waiting for the lock 2026-10-30T00:20:14Z +== H53-smoke-B2-a lock taken 2026-10-30T00:20:14Z +== H53-smoke-B2-a start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=B2 LIVE_CAP=2370 LIVE_HOLD=2 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER= LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold 2026-10-30T00:20:14Z +== H53-smoke-B2-a rc=143; clusters deleted: none; containers removed: none; now: 2026-10-30T00:25:26Z +== H53-smoke-B2-b waiting for the lock 2026-10-30T00:25:35Z +== H53-smoke-B2-b lock taken 2026-10-30T00:25:35Z +== H53-smoke-B2-b start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=B2 LIVE_CAP=2370 LIVE_HOLD=2 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER=b LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold 2026-10-30T00:25:35Z +== H53-smoke-B2-b rc=143; clusters deleted: none; containers removed: none; now: 2026-10-30T00:36:28Z diff --git a/bench/eatflow/results/2026-10-20/H53/window/deadman.log b/bench/eatflow/results/2026-10-20/H53/window/deadman.log new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/window/events.log b/bench/eatflow/results/2026-10-20/H53/window/events.log new file mode 100644 index 00000000..166a365c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/events.log @@ -0,0 +1,23 @@ +2026-10-30T00:14:50Z VRAM before the stop: 0000:12:00.0 16.14;0000:48:00.0 15.43;0000:7c:00.0 16.18;0000:96:00.0 15.47; +2026-10-30T00:15:00Z qwen38-a-hr stopped +2026-10-30T00:15:06Z qwen38-b-hr stopped +2026-10-30T00:15:06Z window open; deadman pid 1548781 +2026-10-30T00:15:06Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.17;0000:96:00.0 0.0; +2026-10-30T00:15:06Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.17;0000:96:00.0 0.0; +2026-10-30T00:15:13Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:15:13Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:15:13Z smoke H53-smoke-AG start +2026-10-30T00:20:14Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:20:14Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:20:14Z smoke H53-smoke-B2-a start +2026-10-30T00:25:19Z smoke H53-smoke-B2-a early stop: setvars.sh missing, 2 restarts (R1 item 4); TERM to hold.sh pid 1565186 +2026-10-30T00:25:35Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:25:35Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:25:35Z smoke H53-smoke-B2-b start +2026-10-30T00:36:21Z smoke H53-smoke-B2-b early stop: oneCCL init_device_fds opendir failed (/dev/dri/by-path absent in the k3d node and pod), the same error at 00:33:38Z and 00:35:27Z (R1 item 4); TERM to hold.sh pid 1578915 +2026-10-30T00:36:38Z B2 ladder: rung (b) fails with an error no further rung addresses ((c) --disable-log-requests is absent; (d) is for a KV-cache failure): B2 not deliverable, no hold runs (H53 pre-registration item 3, R1 item 4); restoring serving +2026-10-30T00:37:45Z qwen38-a-hr running=true; qwen38-a-hr: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:45Z qwen38-a-hr: image, command, env, devices, mounts and ports as before the stop +2026-10-30T00:37:47Z qwen38-b-hr running=true; qwen38-b-hr: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:47Z qwen38-b-hr: image, command, env, devices, mounts and ports as before the stop +2026-10-30T00:38:03Z window closed (restore rc=0) diff --git a/bench/eatflow/results/2026-10-20/H53/window/holds-H53.log b/bench/eatflow/results/2026-10-20/H53/window/holds-H53.log new file mode 100644 index 00000000..e58209b1 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/holds-H53.log @@ -0,0 +1,6 @@ +{"ev": "start", "host": "905a08313800", "name": "H53-smoke-AG", "t": 1793319313.776, "tag": "H53-smoke-AG:1549024:380698146"} +{"ev": "end", "host": "905a08313800", "orphans_killed": "0", "rc": "0", "reap_residue": "0", "stale_reaped": "0", "t": 1793319605.845, "tag": "H53-smoke-AG:1549024:380698146"} +{"ev": "start", "host": "905a08313800", "name": "H53-smoke-B2-a", "t": 1793319614.615, "tag": "H53-smoke-B2-a:1565186:380728230"} +{"ev": "end", "host": "905a08313800", "orphans_killed": "0", "rc": "143", "reap_residue": "0", "stale_reaped": "0", "t": 1793319926.825, "tag": "H53-smoke-B2-a:1565186:380728230"} +{"ev": "start", "host": "905a08313800", "name": "H53-smoke-B2-b", "t": 1793319935.15, "tag": "H53-smoke-B2-b:1578915:380760283"} +{"ev": "end", "host": "905a08313800", "orphans_killed": "0", "rc": "143", "reap_residue": "0", "stale_reaped": "0", "t": 1793320588.769, "tag": "H53-smoke-B2-b:1578915:380760283"} diff --git a/bench/eatflow/results/2026-10-20/H53/window/restore.log b/bench/eatflow/results/2026-10-20/H53/window/restore.log new file mode 100644 index 00000000..1950dd11 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/restore.log @@ -0,0 +1,4 @@ +2026-10-30T00:36:38Z qwen38-a-hr started +2026-10-30T00:36:38Z qwen38-b-hr started +2026-10-30T00:37:40Z qwen38-b-hr answering: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:43Z qwen38-a-hr answering: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " diff --git a/bench/eatflow/results/2026-10-20/H53/window/serving-events.log b/bench/eatflow/results/2026-10-20/H53/window/serving-events.log new file mode 100644 index 00000000..5abd3286 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/serving-events.log @@ -0,0 +1,8 @@ +2026-10-30T00:14:52Z qwen38-a-hr before stop: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:15:00Z qwen38-a-hr stopped (ZE_AFFINITY_MASK=0,1, port 30000) +2026-10-30T00:15:01Z qwen38-b-hr before stop: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:15:06Z qwen38-b-hr stopped (ZE_AFFINITY_MASK=2,3, port 30001) +2026-10-30T00:36:38Z qwen38-a-hr started +2026-10-30T00:36:38Z qwen38-b-hr started +2026-10-30T00:37:40Z qwen38-b-hr answering: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:43Z qwen38-a-hr answering: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " diff --git a/bench/eatflow/results/2026-10-20/H53/window/start.launch.log b/bench/eatflow/results/2026-10-20/H53/window/start.launch.log new file mode 100644 index 00000000..12a5704d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/start.launch.log @@ -0,0 +1,7 @@ +2026-10-30T00:37:45Z qwen38-a-hr running=true; qwen38-a-hr: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:45Z qwen38-a-hr: image, command, env, devices, mounts and ports as before the stop +2026-10-30T00:37:47Z qwen38-b-hr running=true; qwen38-b-hr: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:47Z qwen38-b-hr: image, command, env, devices, mounts and ports as before the stop +2026-10-30T00:37:52Z :30000 pass=1 before=2 after=3 content=OK +2026-10-30T00:38:03Z :30001 pass=1 before=3 after=4 content=OK +2026-10-30T00:38:03Z window closed (restore rc=0) diff --git a/bench/eatflow/results/2026-10-20/H53/window/stop.log b/bench/eatflow/results/2026-10-20/H53/window/stop.log new file mode 100644 index 00000000..c509cd0d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/stop.log @@ -0,0 +1,4 @@ +2026-10-30T00:14:52Z qwen38-a-hr before stop: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:15:00Z qwen38-a-hr stopped (ZE_AFFINITY_MASK=0,1, port 30000) +2026-10-30T00:15:01Z qwen38-b-hr before stop: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:15:06Z qwen38-b-hr stopped (ZE_AFFINITY_MASK=2,3, port 30001) diff --git a/bench/eatflow/results/2026-10-20/H53/window/xe-prom.json b/bench/eatflow/results/2026-10-20/H53/window/xe-prom.json new file mode 100644 index 00000000..1017281d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/xe-prom.json @@ -0,0 +1 @@ +{"t0": 1793319300, "t1": 1793320690, "note": "read-only query_range of crescent Prometheus after the window, step 15 s", "xe_vram_used_bytes": [{"metric": {"__name__": "xe_vram_used_bytes", "bdf": "0000:12:00.0", "card": "card1", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "181055488"], [1793320275, "181055488"], [1793320290, "362127360"], [1793320305, "543207424"], [1793320320, "905367552"], [1793320335, "0"], [1793320350, "181051392"], [1793320365, "181051392"], [1793320380, "362131456"], [1793320395, "543223808"], [1793320410, "724287488"], [1793320425, "0"], [1793320440, "0"], [1793320455, "181055488"], [1793320470, "181055488"], [1793320485, "181055488"], [1793320500, "543211520"], [1793320515, "724271104"], [1793320530, "181055488"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "181055488"], [1793320635, "543129600"], [1793320650, "27628863488"], [1793320665, "28720324608"], [1793320680, "28722880512"]]}, {"metric": {"__name__": "xe_vram_used_bytes", "bdf": "0000:48:00.0", "card": "card2", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "2150400"], [1793320275, "2150400"], [1793320290, "183209984"], [1793320305, "364261376"], [1793320320, "547430400"], [1793320335, "0"], [1793320350, "2134016"], [1793320365, "2134016"], [1793320380, "183205888"], [1793320395, "364249088"], [1793320410, "545284096"], [1793320425, "0"], [1793320440, "0"], [1793320455, "2125824"], [1793320470, "2125824"], [1793320485, "2125824"], [1793320500, "185303040"], [1793320515, "545308672"], [1793320530, "2125824"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "2101248"], [1793320635, "185241600"], [1793320650, "15433220096"], [1793320665, "16510111744"], [1793320680, "16512667648"]]}, {"metric": {"__name__": "xe_vram_used_bytes", "bdf": "0000:7c:00.0", "card": "card3", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "17368285184"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "6586368"], [1793319540, "1816489984"], [1793319555, "15901691904"], [1793319570, "15901642752"], [1793319585, "15901642752"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "2105344"], [1793320275, "2105344"], [1793320290, "183152640"], [1793320305, "364199936"], [1793320320, "547348480"], [1793320335, "0"], [1793320350, "2117632"], [1793320365, "2117632"], [1793320380, "183156736"], [1793320395, "364199936"], [1793320410, "545243136"], [1793320425, "0"], [1793320440, "0"], [1793320455, "2113536"], [1793320470, "2113536"], [1793320485, "2113536"], [1793320500, "185266176"], [1793320515, "545251328"], [1793320530, "2113536"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "181043200"], [1793320635, "543141888"], [1793320650, "27856384000"], [1793320665, "28471382016"], [1793320680, "28471840768"]]}, {"metric": {"__name__": "xe_vram_used_bytes", "bdf": "0000:96:00.0", "card": "card4", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "16610652160"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "2113536"], [1793320275, "2113536"], [1793320290, "183156736"], [1793320305, "364199936"], [1793320320, "547348480"], [1793320335, "0"], [1793320350, "2105344"], [1793320365, "2105344"], [1793320380, "183144448"], [1793320395, "364183552"], [1793320410, "545226752"], [1793320425, "0"], [1793320440, "0"], [1793320455, "2109440"], [1793320470, "2109440"], [1793320485, "2109440"], [1793320500, "185253888"], [1793320515, "545230848"], [1793320530, "2109440"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "2105344"], [1793320635, "185241600"], [1793320650, "15646273536"], [1793320665, "16261324800"], [1793320680, "16261787648"]]}], "xe_engine_utilization_percent": [{"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "bcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "1.11"], [1793320665, "12.37"], [1793320680, "3.76"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "ccs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "15"], [1793320665, "24.15"], [1793320680, "9.59"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "rcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "vcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "vecs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "bcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "1.08"], [1793320665, "11.3"], [1793320680, "3.6"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "ccs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "15.96"], [1793320665, "22.18"], [1793320680, "9.58"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "rcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "vcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "vecs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "bcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "1.97"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "45.46"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "1.72"], [1793320665, "9.84"], [1793320680, "2.01"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "ccs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "5.8"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "19.42"], [1793320665, "14.44"], [1793320680, "6"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "rcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "vcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "vecs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "bcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "2.01"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "1.83"], [1793320665, "9.88"], [1793320680, "2.27"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "ccs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "5.8"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "19.05"], [1793320665, "14.53"], [1793320680, "6.02"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "rcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "vcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "vecs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}]} \ No newline at end of file diff --git a/bench/eatflow/solo.py b/bench/eatflow/solo.py index a7ddabcd..88e01df9 100644 --- a/bench/eatflow/solo.py +++ b/bench/eatflow/solo.py @@ -126,6 +126,32 @@ def block_pairs(attempts, solo_rows, a, b): MIN_BLOCKS = 3 +MAX_BLOCKS = 4 # holds 7 and 8 are the last block (no hold runs after eight) + + +def alive_blocks(solo_rows, ran, min_pairs=MIN_BLOCK_PAIRS, max_blocks=MAX_BLOCKS): + """H53 amendment R1's futility rule: the blocks (1..max_blocks) that can still count, given + solo.jsonl's rows and `ran`, the hold numbers that have run (a hold that delivered nothing has + run but has no rows). A block is out when a hold of it has run with fewer than min_pairs + complete attempts (its complete pairs can be no more), or when both of its holds have run and + it has fewer than min_pairs complete pairs. It reads only which blocks may count, never d.""" + status = {(r["hold"], r["j"]): r["status"] for r in solo_rows} + done = {} + for (h, _), s in status.items(): + done[h] = done.get(h, 0) + (s == "complete") + alive = [] + for k in range(1, max_blocks + 1): + h1, h2 = 2 * k - 1, 2 * k + if any(h in ran and done.get(h, 0) < min_pairs for h in (h1, h2)): + continue + if h1 in ran and h2 in ran: + js = {j for (h, j) in status if h == h1} + if sum(status.get((h1, j)) == "complete" and status.get((h2, j)) == "complete" for j in js) < min_pairs: + continue + alive.append(k) + return alive + + def block_verdict(res, a, b, threshold=0.30, min_rate=0.6): """The H53 verdict from pairs.analyze's result over block pairs (key 1000 * k + j). Per block k: d_k = mean over its matched pairs of ok(a) - ok(b). Over the n blocks with at least diff --git a/bench/eatflow/test_pairs.py b/bench/eatflow/test_pairs.py index 09a525cb..bdf95bb2 100644 --- a/bench/eatflow/test_pairs.py +++ b/bench/eatflow/test_pairs.py @@ -888,6 +888,68 @@ def test_live_sh_keeps_the_arm_off_the_serving_cards(self): self.assertLess(i_hold, i_apply) self.assertLess(i_check, i_run) + def test_stage_path_is_defined_before_the_solo_path_calls_it(self): + # H53 amendment R1: stage_path lived inside the pairs branch, so the solo AG path called an + # undefined function (bash went on: no -e) and OVMS would have served an empty hostPath. + live = self.live() + self.assertEqual(live.count("stage_path() {"), 1) + i_def = live.index("\nstage_path() {") # top level, not inside a branch + i_solo = live.index('if [ "${MODE}" = solo ]; then') + self.assertLess(i_def, i_solo) + blk = live[i_solo:live.index('if [ "${MODE}" = pairs ]; then')] + self.assertIn('stage_path "${TGT_DIR}" "${TGT}" 5000', blk) + # Run the definition as bash does, with stub nodes/docker: it copies into every node. + fn = live[i_def + 1:live.index("\n}\n", i_def) + 3] + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + script = ("set -u\nM=/m\nnodes() { echo n1 n2; }\ndocker() { echo \"docker $*\" >> " + tmp + "/calls; }\n" + + fn + 'stage_path Qwen3.8-27B-int4-ov OpenVINO/Qwen3.8-27B-int4-ov 5000\n') + import subprocess + p = subprocess.run(["bash", "-c", script], capture_output=True, text=True) + self.assertEqual(p.returncode, 0, p.stderr) + with open(os.path.join(tmp, "calls")) as f: + calls = f.read() + for n in ("n1", "n2"): + self.assertIn(f"docker cp -L /m/Qwen3.8-27B-int4-ov/. {n}:/var/apb-models/OpenVINO/Qwen3.8-27B-int4-ov/", calls) + self.assertIn(f"docker exec {n} chown -R 5000:5000 /var/apb-models/OpenVINO/Qwen3.8-27B-int4-ov", calls) + + def test_a_smoke_can_set_the_solo_task_guard(self): + live = self.live() + blk = live[live.index('if [ "${MODE}" = solo ]; then'):live.index('if [ "${MODE}" = pairs ]; then')] + self.assertIn('${LIVE_PAIRS_GUARD:+--guard-s "${LIVE_PAIRS_GUARD}"}', blk) + with open(os.path.join(HERE, "crescent", "solo-hold.sh")) as f: + self.assertNotIn("LIVE_PAIRS_GUARD", f.read()) # holds keep agentbench's own guard + + def test_h53_window_steps(self): + p = os.path.join(HERE, "results", "2026-10-20", "H53", "prehold", "prehold.sh") + with open(p) as f: + s = f.read() + # The four cards of both pairs (pre-registration "Cards and serving"), stop and start only. + self.assertIn('FREE="0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0"', s) + self.assertIn('PAIRS="qwen38-a-hr qwen38-b-hr"', s) + self.assertNotIn("docker rm", s) + self.assertNotIn("docker stop", s) # only through gpu-serving.sh + i_inspect, i_stop = s.index('docker inspect "${c}" > "${O}/inspect-${c}.before.json"'), s.index("gpu-serving.sh\" stop") + self.assertLess(i_inspect, i_stop) + # A smoke runs the warm-up only; holds run solo-hold.sh with the registered arm. + self.assertIn("LIVE_PAIRS_GUARD=99999", s) + self.assertIn('bash "${C}/crescent/solo-hold.sh" "${n}" "${arm}"', s) + self.assertIn('arm=$(python3 "${C}/pairs.py" arm H53 "${n}")', s) + # Holds 7 and 8 only when fewer than three blocks count after hold 6. + self.assertIn('[ "${k}" -ge 3 ] || { last=8;', s) + # Every card must be free before a smoke or a hold (polled, Prometheus scrapes every 15 s). + self.assertEqual(s.count("wait_cards ||"), 2) + self.assertIn("FREE_GIB=2", s) + # R1 item 5's futility stop reads solo.alive_blocks after every hold. + i_hold, i_alive = s.index('one_hold "${n}" "${rungs}" ||'), s.index("a=$(alive_blocks)") + self.assertLess(i_hold, i_alive) + self.assertIn('[ "$(wc -w <<< "${a}")" -lt 3 ]', s) + self.assertIn("solo.alive_blocks(rows,", s) + # E4: setup.sh runs from this checkout under the bench lock. + blk = s[s.index("\nsetup)"):s.index("\nverify)")] + self.assertIn("lock", blk) + self.assertIn('bash "${C}/crescent/setup.sh"', blk) + class PrerunPinsTest(unittest.TestCase): """Crescent amendments E1 and E2 (EXPERIMENTS.md, 2026-10-29): every crescent agent-row @@ -931,6 +993,84 @@ def test_h50_and_h51_run_eat_release_110_b1(self): self.assertIn("EAT_PR21=c86c42b073ee533e9fb9e9b9330b9d1478e4bf47", self.read("crescent", "setup.sh")) +class H62Test(unittest.TestCase): + """H62 (EXPERIMENTS.md "H62 pre-registration"): H50's design, statistic and rule (L3) with + arm B5, Intel's EAS default, and both servers in KVM guests. A pod cgroup can be read + through a command ("exec:"), so the idle-arm check reads the pod inside its guest.""" + + REG = {"hyp": "H62", "rule": "L3", "seed": "H62-crescent-2026-10-29"} + + def read_live(self): + with open(os.path.join(HERE, "live.sh")) as f: + return f.read() + + def test_h62_is_h50s_design_and_rule_with_its_own_seed(self): + self.assertEqual(PR.design("H62"), ("H49", "L3")) + self.assertEqual(PR.ORDER_SEEDS["H62"], self.REG["seed"]) + self.assertIn("H62", AB.PAIRS_HYPS) + at = [dict(r, arm="B5") if r["arm"] == "B1" else r for r in L2Test.h49_like(L2Test())] + res = PR.analyze(at, pairs_of(24), "H62", "A1", "B5", registration=self.REG) + ref = PR.analyze(L2Test.h49_like(L2Test()), pairs_of(24), "H50", "A1", "B1", + registration={"hyp": "H50", "rule": "L3", "seed": PR.ORDER_SEEDS["H50"]}) + self.assertEqual((res["rule"], res["arms"], res["verdict"]), ("L3", ["A1", "B5"], "SUPPORTED")) + self.assertEqual(res["paired_diff"], ref["paired_diff"]) # H50's statistic, unchanged + self.assertEqual(res["confidence"]["support_share"], ref["confidence"]["support_share"]) + with self.assertRaises(ValueError): # H50's registration is not H62's + PR.analyze(at, pairs_of(24), "H62", "A1", "B5", + registration={"hyp": "H50", "rule": "L3", "seed": PR.ORDER_SEEDS["H50"]}) + + def test_exec_cgroup_reads_a_command(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + fake = os.path.join(tmp, "arm.sh") + with open(fake, "w") as f: + f.write('#!/bin/sh\n[ "$1 $2 $3" = "A1 cpustat /sys/fs/cgroup/x" ] || exit 7\n' + 'printf "usage_usec 9001\\nuser_usec 5\\n"\n') + os.chmod(fake, 0o755) + self.assertEqual(PR.cgroup_usage_usec(f"exec:{fake} A1 cpustat /sys/fs/cgroup/x"), 9001) + self.assertIsNone(PR.cgroup_usage_usec(f"exec:{fake} A1 cpustat /elsewhere")) # non-zero exit + self.assertIsNone(PR.cgroup_usage_usec(f"exec:{os.path.join(tmp, 'missing')}")) + self.assertIsNone(PR.cgroup_usage_usec("exec:")) + # A probe snapshot carries the remote reading like a local one; the window uses it. + probe = PR.Probe({"A1": f"exec:{fake} A1 cpustat /sys/fs/cgroup/x"}, [], rapl=False) + self.assertEqual(list(probe.snap()["pods"].values()), [9001]) + # `pairs.py usage SPEC...` reads each spec once: what live.sh checks before a hold. + import subprocess + ok = subprocess.run([sys.executable, os.path.join(HERE, "pairs.py"), "usage", + f"exec:{fake} A1 cpustat /sys/fs/cgroup/x"], capture_output=True, text=True) + self.assertEqual(ok.returncode, 0, ok.stderr) + self.assertIn("9001", ok.stdout) + bad = subprocess.run([sys.executable, os.path.join(HERE, "pairs.py"), "usage", + f"exec:{fake} A1 cpustat /sys/fs/cgroup/x", f"exec:{fake} B5 cpustat /y"], + capture_output=True, text=True) + self.assertEqual(bad.returncode, 1) + + def test_live_sh_runs_h62_against_remote_arms(self): + live = self.read_live() + self.assertIn('[ "${ARM}" = H62 ] && DESIGN=H62', live) + self.assertRegex(live, r"\n\s*H62\) A=A1 B=B5 ;;") + pairs = live[live.index('if [ "${MODE}" = pairs ]; then'):] + remote = pairs[pairs.index('if [ "${DESIGN}" = H62 ]; then'):pairs.index("\n else\n")] + # Neither arm is deployed on this host: no cluster, no staging, no local port-forward. + for local in ("cluster_up", "stage_path", "kubectl ", "agent manifest", "helm template"): + self.assertNotIn(local, remote, local) + for step in ('"${RA}" restart "${left}"', '"${RB}" restart "${left}"', 'records "${OUT}/remote-a"', + 'records "${OUT}/remote-b"', '"${RA}" forward', '"${RB}" forward', 'P_SRV=$("${RA}" port)', + 'BPORT=$("${RB}" port)', 'cg_a=$("${RA}" cgroup)', 'cg_b=$("${RB}" cgroup)', + 'python3 "${HERE}/pairs.py" usage "${cg_a}" "${cg_b}"'): + self.assertIn(step, remote, step) + self.assertIn("left=$(( CAP - $(el) - 900 ))", remote) # the 900 s readiness rule + self.assertIn("NOT DELIVERABLE: not ready with 900 s of the hold left", remote) + # The local arms run in the else branch unchanged, and the harness after it is shared. + local = pairs[pairs.index("\n else\n"):pairs.index(" litellm() {")] + self.assertIn(" cluster_up\n", local) + self.assertIn("\n fi\n", local[-40:]) + # The guests' port-forwards stop with the hold. + clean = live[live.index("cleanup() {"):live.index("trap cleanup EXIT")] + self.assertIn('for r in "${REMOTE[@]}"; do "${r}" unforward >/dev/null 2>&1; done', clean) + self.assertIn("REMOTE=()", live[:live.index("cleanup() {")]) + + class B3RecordsTest(unittest.TestCase): """H50o amendment O1 item 2: the records of B3's pulled model that the pre-registration names (the init container's log, every file's sha256, the graph each hold serves), read from the diff --git a/bench/eatflow/test_solo.py b/bench/eatflow/test_solo.py index 8d4423cf..4cd97471 100644 --- a/bench/eatflow/test_solo.py +++ b/bench/eatflow/test_solo.py @@ -199,6 +199,32 @@ def test_a_short_block_is_not_counted(self): self.assertEqual((bv["n_blocks"], bv["verdict"]), (2, "INCONCLUSIVE")) +class AliveBlocksTest(unittest.TestCase): + """H53 amendment R1's futility rule: which blocks can still count.""" + + @staticmethod + def rows(hold, n, void=()): + return [{"hold": hold, "j": j, "status": "void" if j in void else "complete"} for j in range(n)] + + def test_nothing_run_leaves_every_block(self): + self.assertEqual(S.alive_blocks([], set()), [1, 2, 3, 4]) + + def test_a_hold_that_delivered_nothing_takes_its_block_out(self): + # Hold 1 ran (rc recorded) with no rows: block 1 cannot reach 8 pairs; 2-4 still can. + self.assertEqual(S.alive_blocks([], {1}), [2, 3, 4]) + self.assertEqual(S.alive_blocks(self.rows(2, 24), {1, 2}), [2, 3, 4]) + + def test_a_short_hold_and_a_short_block(self): + # Hold 3 ran 7 complete attempts: block 2 is out. Holds 5 and 6 ran 10 each but only + # 7 tasks overlap complete (hold 6 voided three): block 3 is out. + rows = self.rows(1, 12) + self.rows(2, 12) + self.rows(3, 7) + self.rows(5, 10) + self.rows(6, 10, void={0, 1, 2}) + self.assertEqual(S.alive_blocks(rows, {1, 2, 3, 5, 6}), [1, 4]) + + def test_two_blocks_out_leave_fewer_than_three(self): + rows = self.rows(2, 24) + self.rows(3, 24) + self.assertEqual(len(S.alive_blocks(rows, {1, 2, 3, 4})), 2) # holds 1 and 4 (AG) delivered nothing + + class RunSoloTest(unittest.TestCase): def setUp(self): self.tmp = tempfile.mkdtemp() diff --git a/bench/h54/results/2026-10-29/H54d/not-run.txt b/bench/h54/results/2026-10-29/H54d/not-run.txt new file mode 100644 index 00000000..9ad8c807 --- /dev/null +++ b/bench/h54/results/2026-10-29/H54d/not-run.txt @@ -0,0 +1,26 @@ +H54d: re-checked, still not run, no hold used (2026-10-29 PDT, 2026-10-30T00:55Z). +Verdict unchanged: INCONCLUSIVE (not run). Supersedes reason 3 of 2026-10-21/H54d/not-run.txt +(H54q has since run, 2026-10-29, 3 holds). Reasons 1 and 2 stand. + +1. A* is undefined. H54a is closed (3 of 3 holds spent, R3's carry used). Its rule picks the + fastest arm among those with GSM8K >= B1g's score - 5/50. B1g has no score, so score-a chose + none. H54q's 200-item reference R is not part of H54a's rule. +2. B1g stays EAT 0d3438d's vLLM CPU v0.24.0 (amendment E1: "H54a and H54d keep B1g at v0.24.0; + both are closed"). Note N2: the stall is vllm-project/vllm#45844, absent from v0.24.0; all three + v0.24.0 Gemma engines on crescent stalled within their first 690-1,540 decode steps. Under + H50's procedure that trips the idle-arm check and quiesce wait, and a warm-up stall stops the + hold before any pair. +3. New: an amendment defining A* and moving B off v0.24.0 would now be written after the H50 run + on the same flow (E1's B1, vLLM CPU v0.28.0: 21/24 within budget; A1 19/24; REFUTED), i.e. + with the moved B's outcome measured, and after H54a's decode rates and H54q's scores. It would + not be a pre-registration, so none is written. +4. New: amendment Q1 moved 2 of H54d's 3 holds to H54q. H54 campaign holds used: 9 of 10 + (H54a 3, H54q 3, H54b 2, H54c 1). H54d holds the last one. +5. Runner: `bench/eatflow/pairs.py design H54d` -> "H54d: not a pairs hypothesis"; live.sh + pairs mode deploys A1 (OVMS) as arm A in every design. + +Checked on crescent at 2026-10-30T00:48-00:50Z (read only): no H54d entry in any campaign.log or +holds.log; ~/k3dgpu-gemma/results/H54d holds only the 2026-10-21 not-run.txt; bench lock free +(inode not in /proc/locks), no k3d cluster, / at 31%. No container, image, file or lock was +touched. Serving pairs not stopped; one front-door chat check through :8100 at 00:50:15Z +returned "OK" four times, sglang:num_requests_total rose on :30000 (7 -> 8) and :30001 (5 -> 8). diff --git a/bench/h62/arm-A1 b/bench/h62/arm-A1 new file mode 100755 index 00000000..3bac3c89 --- /dev/null +++ b/bench/h62/arm-A1 @@ -0,0 +1,3 @@ +#!/usr/bin/env bash +# arm-A1: bench/h62/arm.sh for arm A1, as one word (live.sh's LIVE_REMOTE_A). +exec "$(dirname "$0")/arm.sh" A1 "$@" diff --git a/bench/h62/arm-B5 b/bench/h62/arm-B5 new file mode 100755 index 00000000..5b9c4b02 --- /dev/null +++ b/bench/h62/arm-B5 @@ -0,0 +1,3 @@ +#!/usr/bin/env bash +# arm-B5: bench/h62/arm.sh for arm B5, as one word (live.sh's LIVE_REMOTE_B). +exec "$(dirname "$0")/arm.sh" B5 "$@" diff --git a/bench/h62/arm.sh b/bench/h62/arm.sh new file mode 100755 index 00000000..4ee85b17 --- /dev/null +++ b/bench/h62/arm.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# arm.sh ARM CMD [ARG]: one H62 arm's server in its KVM guest (EXPERIMENTS.md "H62 +# pre-registration"). bench/eatflow/live.sh's remote-arm mode calls it through arm-A1 and arm-B5. +# port the host port its Service is forwarded to (the guest's hostfwd) +# restart SECONDS delete its serving pod, then wait until a new pod is Ready; exit 3 when none is +# Ready within SECONDS (the hold's 900 s readiness rule passes the time left) +# records DIR the pod, the CPU ISA it sees (amx_tile lines), its cpuset and nproc, the nodes, +# the guest's load and memory, and B5's LLMInferenceService +# logs DIR every pod, events, the pod description and the server log (a hold that fails) +# forward (re)start `kubectl port-forward` of its Service in the guest, on the serve port +# unforward stop that port-forward +# cgroup "exec: ARM cpustat ": its pod's cgroup in the guest, as the +# idle-arm check reads it (bench/eatflow/pairs.py) +# cpustat DIR that cgroup's cpu.stat +# ARM: A1 (VM a: k3d-gpu's agent OVMS) or B5 (VM b: EAS's gemma-4-26b LLMInferenceService). +# H62_STATE (default ~/k3dgpu-bench/h62) holds ssh_config; H62_SSH is the ssh client (tests stub +# it); H62_POLL_S is the restart poll period (default 5 s). Each guest's ~/h62/env sets PATH and +# KUBECONFIG for kubectl there. +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +SELF=${HERE}/arm.sh +P() { python3 "${HERE}/h62.py" "$@"; } +ARM=${1:?usage: arm.sh A1|B5 CMD [ARG]} CMD=${2:?usage: arm.sh A1|B5 CMD [ARG]} +fields=$(P arm-fields "${ARM}" 2>/dev/null) || { echo "arm.sh: no arm ${ARM} (A1 or B5)" >&2; exit 2; } +read -r _ HOST PORT NS SEL SVC SPORT CTR EXTRA <<< "${fields}" +STATE=${H62_STATE:-${HOME}/k3dgpu-bench/h62} +# g CMD: one command in the guest, after its ~/h62/env. +g() { ${H62_SSH:-ssh} -F "${STATE}/ssh_config" "${HOST}" ". ~/h62/env && $*"; } +pods() { g "kubectl -n ${NS} get pods -l ${SEL} -o json"; } +ready_pod() { pods | P pick-pod "${1:-}"; } # " " of a Ready pod whose uid is not in $1 +case ${CMD} in +port) + echo "${PORT}" ;; +restart) + left=${3:-} + [ -n "${left}" ] || { echo "usage: arm.sh ${ARM} restart SECONDS" >&2; exit 2; } + [ "${left}" -gt 0 ] 2>/dev/null || { echo "no time left to restart"; exit 3; } + end=$(( SECONDS + left )) + old=$(pods | P uids) || { echo "cannot list the pods"; exit 1; } + g "kubectl -n ${NS} delete pod -l ${SEL} --wait=false" || { echo "cannot delete the pods"; exit 1; } + while [ "${SECONDS}" -lt "${end}" ]; do + if p=$(ready_pod "${old}"); then echo "ready: ${p}"; exit 0; fi + sleep "${H62_POLL_S:-5}" + done + echo "not ready within ${left} s" + exit 3 ;; +records) + d=${3:?usage: arm.sh ARM records DIR} + mkdir -p "${d}" + read -r name uid <<< "$(ready_pod "")" + [ -n "${name:-}" ] || { echo "no Ready pod"; exit 1; } + echo "${name} ${uid}" > "${d}/pod.txt" + g "kubectl -n ${NS} get pod ${name} -o json" > "${d}/pod.json" + g "kubectl -n ${NS} exec ${name} -c ${CTR} -- grep -c amx_tile /proc/cpuinfo" > "${d}/isa.txt" 2>&1 + g "kubectl -n ${NS} exec ${name} -c ${CTR} -- sh -c 'cat /sys/fs/cgroup/cpuset.cpus.effective; nproc'" \ + > "${d}/cpuset.txt" 2>&1 + g "kubectl get nodes -o json" > "${d}/nodes.json" + g "uptime; free -g; grep -c amx_tile /proc/cpuinfo; nproc" > "${d}/guest.txt" 2>&1 + [ -z "${EXTRA}" ] || g "kubectl -n ${NS} get ${EXTRA} -o yaml" > "${d}/extra.yaml" 2>&1 + g "kubectl get pods -A -o wide" > "${d}/pods.txt" 2>&1 + echo "records: ${d}" ;; +logs) + d=${3:?usage: arm.sh ARM logs DIR} + mkdir -p "${d}" + g "kubectl get pods -A -o wide" > "${d}/pods.txt" 2>&1 + g "kubectl get events -A --sort-by=.lastTimestamp" > "${d}/events.txt" 2>&1 + g "kubectl -n ${NS} describe pods -l ${SEL}" > "${d}/describe.txt" 2>&1 + g "kubectl -n ${NS} logs -l ${SEL} -c ${CTR} --tail=1000" > "${d}/server.log" 2>&1 + echo "logs: ${d}" ;; +forward) + g "pid=\$(cat ~/h62/pf-${ARM}.pid 2>/dev/null) && kill \${pid} 2>/dev/null;" \ + "setsid nohup kubectl -n ${NS} port-forward --address 0.0.0.0 ${SVC} ${PORT}:${SPORT}" \ + "> ~/h62/pf-${ARM}.log 2>&1 < /dev/null & echo \$! > ~/h62/pf-${ARM}.pid" ;; +unforward) + g "pid=\$(cat ~/h62/pf-${ARM}.pid 2>/dev/null) && kill \${pid} 2>/dev/null; rm -f ~/h62/pf-${ARM}.pid" ;; +cgroup) + read -r _ uid <<< "$(ready_pod "")" + [ -n "${uid:-}" ] || { echo "no Ready pod" >&2; exit 1; } + # cgroupfs driver (k3s in VM a's k3d node): pod; systemd driver (Kubespray's containerd + # in VM b): ...-pod.slice. + dir=$(g "find /sys/fs/cgroup -maxdepth 9 -type d \\( -name pod${uid} -o -name '*pod${uid//-/_}.slice' \\) -print -quit") + [ -n "${dir}" ] || { echo "no cgroup for pod ${uid}" >&2; exit 1; } + echo "exec:${SELF} ${ARM} cpustat ${dir}" ;; +cpustat) + g "cat ${3:?usage: arm.sh ARM cpustat DIR}/cpu.stat" ;; +*) + echo "usage: arm.sh A1|B5 port|restart SECONDS|records DIR|logs DIR|forward|unforward|cgroup|cpustat DIR" >&2 + exit 2 ;; +esac diff --git a/bench/h62/campaign.sh b/bench/h62/campaign.sh new file mode 100755 index 00000000..5f7f7034 --- /dev/null +++ b/bench/h62/campaign.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# campaign.sh: H62 end to end on crescent (EXPERIMENTS.md "H62 pre-registration"), from the H62 +# checkout at the deploy commit: setup.sh, smoke.sh, then holds (pairs-hold.sh 1, 2, ...) until +# B24's 24 pairs are recorded or 8 holds have run, then the score. teardown.sh runs at the end, +# also after an abort. +# - setup exit 6 (the host gate still failed after 60 min): the two qwen38 serving pairs are +# stopped under the box owner's grant (gpu-serving.sh stop, never rm) and setup runs once more; +# teardown.sh starts them again +# - a hold that exits 3 (a server not ready with 900 s left) counts as one of the 8 and runs no +# pair; after 2 such holds in a row no further hold runs (INCONCLUSIVE, not deliverable) +# Records: ~/k3dgpu-bench/results/H62 (campaign.log, setup/, pairs/, hold/, report.txt, vmtax.json). +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +B=${HOME}/k3dgpu-bench +R=${B}/results/H62 +MAX_HOLDS=8 NOT_DELIVERABLE_MAX=2 +mkdir -p "${R}" +log() { echo "== campaign $* $(date -u +%FT%TZ)" | tee -a "${R}/campaign.log"; } +teardown() { bash "${HERE}/teardown.sh" >> "${R}/teardown.log" 2>&1; log "teardown exit $?"; } +trap teardown EXIT +log "start at $(git -C "${REPO}" rev-parse HEAD)" +bash "${HERE}/setup.sh" +rc=$? +if [ "${rc}" = 6 ]; then + log "host gate failed: stopping the qwen38 serving pairs (owner's grant; docker stop, never rm)" + for c in qwen38-a-hr qwen38-b-hr; do + if bash "${REPO}/bench/eatflow/crescent/gpu-serving.sh" stop "${c}" >> "${R}/serving.log" 2>&1; then + echo "${c}" >> "${R}/serving-stopped.txt" + fi + done + bash "${HERE}/setup.sh" + rc=$? +fi +[ "${rc}" = 0 ] || { log "setup exit ${rc}: no hold"; exit "${rc}"; } +bash "${HERE}/smoke.sh" +rc=$? +[ "${rc}" = 0 ] || { log "wiring smoke failed (exit ${rc}): no hold"; exit 9; } +nd=0 +for n in $(seq 1 "${MAX_HOLDS}"); do + left=$(python3 "${HERE}/h62.py" pairs-left "${R}/pairs") + [ "${left}" -gt 0 ] || break + log "hold ${n}: ${left} pair(s) left" + bash "${HERE}/pairs-hold.sh" "${n}" + rc=$? + log "hold ${n} exit ${rc}" + if [ "${rc}" = 3 ]; then + nd=$(( nd + 1 )) + [ "${nd}" -lt "${NOT_DELIVERABLE_MAX}" ] || { log "${nd} holds in a row not deliverable: no further hold"; break; } + else + nd=0 + fi +done +python3 "${REPO}/bench/eatflow/agentbench.py" pairs-report "${R}/pairs" --hyp H62 --arm-a A1 --arm-b B5 \ + > "${R}/report.txt" 2>&1 +log "pairs-report exit $?" +h50=${REPO}/bench/eatflow/results/2026-10-20/H50/pairs/analysis.json +if [ -f "${R}/pairs/analysis.json" ]; then + python3 "${HERE}/h62.py" vmtax "${h50}" "${R}/pairs/analysis.json" > "${R}/vmtax.json" +fi diff --git a/bench/h62/guest-a.sh b/bench/h62/guest-a.sh new file mode 100755 index 00000000..7388701c --- /dev/null +++ b/bench/h62/guest-a.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# guest-a.sh [deploy]: VM a's setup for H62's A1 (EXPERIMENTS.md "H62 pre-registration"), run in +# the guest by setup.sh as user ubuntu, under its 45 min ceiling. A1 is H50's A1 unchanged: +# 1. Docker CE from Docker's apt repository at the pinned version (its key's fingerprint +# checked first) and python3-yaml for bench/apb/stage.py; then this script runs again in a +# shell with the docker group, as `deploy` +# 2. A1's IR from the host's read-only share (virtfs tag h62share), checked against its +# manifest.json +# 3. `k3d-gpu up` at the deploy commit (~/h62/k3d-gpu, copied by setup.sh), with H50's node image +# and crescent amendment C1's kubelet eviction thresholds, cluster k3dgpu-e-h62 +# 4. `k3d-gpu agent manifest` with live.sh's gemma4 settings for A1 (CPU, the snapped QAT IR, +# tool parser gemma4, no reasoning parser, no draft, no autoquant), staged and applied as +# live.sh does, and rolled out +# 5. records: amendment A4's `k3d-gpu plan` line, versions, the node, the pod +# k3d and kubectl are setup.sh's checked binaries in ~/h62/bin. Records: ~/h62/records. +set -euo pipefail +G=${HOME}/h62 +P() { python3 "${G}/h62.py" "$@"; } +R=${G}/records +REPO=${G}/k3d-gpu +mkdir -p "${R}" +export PATH=${G}/bin:${PATH} KUBECONFIG=${G}/kubeconfig XDG_STATE_HOME=${G}/state K3D_GPU_CLUSTER=k3dgpu-e-h62 +step() { echo "== $1 $(date -u +%FT%TZ)" | tee -a "${R}/steps.txt"; } +if [ "${1:-}" != deploy ]; then + step docker + # ${PATH} expands when a command in the guest sources ~/h62/env (bench/h62/arm.sh). + # shellcheck disable=SC2016 + printf 'export PATH=%s/bin:${PATH} KUBECONFIG=%s/kubeconfig\n' "${G}" "${G}" > "${G}/env" + curl -fsSLo "${G}/docker.asc" https://download.docker.com/linux/ubuntu/gpg + gpg --show-keys --with-colons "${G}/docker.asc" | awk -F: '/^fpr/ {print $10}' | grep -x "$(P pin docker_gpg_fpr)" > /dev/null \ + || { echo "Docker's apt key is not $(P pin docker_gpg_fpr)"; exit 1; } + sudo install -m 0755 -d /etc/apt/keyrings + sudo install -m 0644 "${G}/docker.asc" /etc/apt/keyrings/docker.asc + echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable" \ + | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + sudo apt-get update -q 2>&1 | tee "${R}/apt-update.log" > /dev/null + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q \ + docker-ce="$(P pin docker_ce)" docker-ce-cli="$(P pin docker_ce)" containerd.io="$(P pin containerd)" \ + python3-yaml="$(P pin python3_yaml)" 2>&1 | tee "${R}/apt-install.log" > /dev/null + sudo usermod -aG docker "$(id -un)" + exec sg docker -c "bash ${G}/guest-a.sh deploy" +fi +step ir +sudo mkdir -p /mnt/h62-ir +mountpoint -q /mnt/h62-ir || sudo mount -t 9p -o trans=virtio,version=9p2000.L,ro h62share /mnt/h62-ir +P ir-verify /mnt/h62-ir > "${R}/ir-verify.txt" +step up +K3D_GPU_IMAGE="$(P pin node_image)" \ + K3D_GPU_K3S_ARGS="--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1%" \ + timeout 900 "${REPO}/scripts/k3d-gpu" up > "${R}/up.log" 2>&1 +step manifest +K3D_GPU_AGENT_DEVICE=CPU K3D_GPU_AGENT_MODEL=k3dgpu/gemma-4-26B-A4B-it-qat-int4-ov K3D_GPU_AGENT_MODEL_NAME=gemma-4-26b-a4b-it \ + K3D_GPU_AGENT_DRAFT=off K3D_GPU_AGENT_TOOL_PARSER=gemma4 K3D_GPU_AGENT_REASONING_PARSER='' K3D_GPU_AUTOQUANT=off \ + "${REPO}/scripts/k3d-gpu" agent manifest > "${R}/agent.yaml" 2> "${R}/agent.err" +python3 "${REPO}/bench/apb/stage.py" ovms "${R}/agent.yaml" "${R}/deploy-a.yaml" +step stage +# As live.sh's stage_path: the IR copied into the node at the path the staged manifest mounts. +node=$(docker ps --filter "label=k3d.cluster=${K3D_GPU_CLUSTER}" --filter label=k3d.role=server --format '{{.Names}}' | head -1) +[ -n "${node}" ] || { echo "no k3d server node"; exit 1; } +t=/var/apb-models/k3dgpu/gemma-4-26B-A4B-it-qat-int4-ov +docker exec "${node}" mkdir -p "${t}" /var/apb-models/.ov_cache +docker cp -L /mnt/h62-ir/. "${node}:${t}/" +docker exec "${node}" rm -rf "${t}/.cache" "${t}/.git" +docker exec "${node}" chown -R 5000:5000 "${t}" +docker exec "${node}" chown 5000:5000 /var/apb-models/.ov_cache +docker exec "${node}" chmod 1777 /var/apb-models +step apply +kubectl apply -f "${R}/deploy-a.yaml" > "${R}/apply.log" +kubectl -n k3d-gpu-agent rollout status deploy/ovms --timeout=1500s > "${R}/rollout.log" 2>&1 +step records +{ timeout 600 "${REPO}/scripts/k3d-gpu" plan --device cpu --model google/gemma-4-26B-A4B-it --explain; echo "rc=$?"; } \ + > "${R}/plan.txt" 2>&1 || true +{ docker version; k3d version; kubectl version --client; uname -r; } > "${R}/versions.txt" 2>&1 +kubectl get nodes -o json > "${R}/nodes.json" +kubectl -n k3d-gpu-agent get pods -o json > "${R}/pods.json" +docker exec "${node}" sh -c "cd ${t} && sha256sum ./*.xml ./*.bin" > "${R}/staged-ir.txt" +step finished diff --git a/bench/h62/guest-b.sh b/bench/h62/guest-b.sh new file mode 100755 index 00000000..2fb70ad0 --- /dev/null +++ b/bench/h62/guest-b.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# guest-b.sh: VM b's setup for H62's B5 (EXPERIMENTS.md "H62 pre-registration"), run in the guest +# by setup.sh as user ubuntu, under its 120 min ceiling. B5 is Intel's documented single-node +# install of the Agent Toolkit: EAT release-2.0.0 docs/single-node-deployment.md from Step 1, with +# the clone the v1.2.0-rc.1 release notes give in place of the guide's `-b release-1.2`, on the +# inventory EAS ships (Option A, localhost: Kubespray installs Kubernetes here). +# - prerequisites.md "Node Kernel Settings", verbatim +# - clone, D1 (the layer manifest's two branch revs pinned to the commits Intel's comments on +# those lines name, before init), configure, init agent-toolkit +# - the documented edit (gemma-4-26b in llm_services_autodeploy_models) and D2 (gemma-4-26b's +# model_id is the QAT checkpoint), then install agent-toolkit +# - wait for the gemma-4-26b workload pod, then the records +# Nothing else is changed: guardrail, hybrid router, observability, NRI balloons, Istio, MetalLB +# and the agent sandbox are EAS's and EAT's defaults. Records: ~/h62/records (no kubeconfig, no +# secret file). +set -euo pipefail +G=${HOME}/h62 +P() { python3 "${G}/h62.py" "$@"; } +R=${G}/records +E=${HOME}/enterprise-ai-solutions +TAG=$(P pin eas_tag) SHA=$(P pin eas_sha) +mkdir -p "${R}" +step() { echo "== $1 $(date -u +%FT%TZ)" | tee -a "${R}/steps.txt"; } +echo "HF_TOKEN: $([ -n "${HF_TOKEN:-}" ] && echo set || echo unset)" > "${R}/hf-token.txt" +step sysctl +sudo tee /etc/sysctl.d/90-inotify.conf > /dev/null <<'EOF' +fs.inotify.max_user_instances = 8192 +fs.inotify.max_user_watches = 1048576 +EOF +sudo sysctl --system 2>&1 | tee "${R}/sysctl.txt" > /dev/null +step clone +[ -d "${E}" ] || git clone -q -b "${TAG}" https://github.com/intel/enterprise-ai-solutions.git "${E}" +cd "${E}" +[ "$(git rev-parse HEAD)" = "${SHA}" ] || { echo "EAS ${TAG} is not ${SHA}"; exit 1; } +step d1 +python3 "${G}/h62.py" eas-pin configs/repos/repos.agent-toolkit.yaml > "${R}/d1.diff" +step configure +# --force answers configure's one y/N prompt (es_auto_installer.sh usage: "required in CI, where +# there is no terminal"); configure's own steps are unchanged. +./es_auto_installer.sh configure --force > "${R}/configure.log" 2>&1 +step init +./es_auto_installer.sh init agent-toolkit > "${R}/init.log" 2>&1 +for pair in "enterprise.ai-inference $(P pin ei_sha)" "enterprise.agent-toolkit $(P pin eat_sha)"; do + read -r dest want <<< "${pair}" + have=$(git -C "ext/${dest}" rev-parse HEAD) + echo "${dest} ${have}" >> "${R}/ext-commits.txt" + [ "${have}" = "${want}" ] || { echo "ext/${dest} is at ${have}, not ${want}"; exit 1; } +done +step edit +# base_domain_name stays EAS's default: global_config.yaml is not edited +python3 "${G}/h62.py" eas-edit env/local > "${R}/edit.diff" +for f in global_config.yaml config.agent-toolkit.yaml config.inference.yaml models.yaml inventory/hosts.yaml; do + [ ! -f "env/local/${f}" ] || cp "env/local/${f}" "${R}/env-${f//\//-}" +done +step install +./es_auto_installer.sh install agent-toolkit > "${R}/install.log" 2>&1 +step ready +export KUBECONFIG=${E}/env/local/kubeconfig.yaml +printf 'export KUBECONFIG=%s\n' "${KUBECONFIG}" > "${G}/env" +NS=$(P arm B5 ns) SEL=$(P arm B5 selector) +until kubectl -n "${NS}" get pods -l "${SEL}" -o json | P pick-pod "" > "${R}/ready-pod.txt"; do sleep 15; done +step records +{ ./es_auto_installer.sh status; echo "rc=$?"; } > "${R}/status.txt" 2>&1 || true +{ python3 --version; yq --version; kubectl version; helm version; uname -r; } > "${R}/versions.txt" 2>&1 || true +kubectl get nodes -o json > "${R}/nodes.json" +kubectl get pods -A -o wide > "${R}/pods.txt" +kubectl -n "${NS}" get pods -o json > "${R}/llm-inference-pods.json" +kubectl -n "${NS}" get llminferenceservices -o yaml > "${R}/llminferenceservices.yaml" +step finished diff --git a/bench/h62/h62.py b/bench/h62/h62.py new file mode 100644 index 00000000..58529914 --- /dev/null +++ b/bench/h62/h62.py @@ -0,0 +1,548 @@ +#!/usr/bin/env python3 +"""h62.py: the pure parts of the H62 runner (EXPERIMENTS.md "H62 pre-registration"): k3d-gpu's +CPU stack A1 against B5, Intel's documented single-node default (EAS v1.2.0-rc.1 installing +EAT 2.0), each in its own KVM guest on crescent. Python 3 standard library only, so the same +file runs on the host and in either guest. + + h62.py pin NAME one pin of PINS (the shell scripts read every pin here) + h62.py vm VM FIELD a field of VMS (VM a or b) + h62.py arm ARM FIELD a field of ARMS (A1 or B5) + h62.py arm-fields ARM every field arm.sh reads, on one line + h62.py ssh-config STATE the ssh_config of both guests + h62.py user-data VM PUBKEY_FILE the guest's cloud-init user-data + h62.py meta-data VM its cloud-init meta-data + h62.py qemu-argv VM STATE IMAGE UID GID KVM_GID [SHARE] + the `docker run` of the guest's QEMU, one argument per line + h62.py host-facts the host gate's facts (MemAvailable, / used, reachability), JSON + h62.py gates PHASE FACTS_JSON PASS/FAIL lines of one gate phase (host, guest, nfd, + weights); exit 1 on any FAIL + h62.py ir-verify DIR A1's IR against its manifest.json; exit 1 on any mismatch + h62.py eas-pin MANIFEST deviation D1, in place: the agent-toolkit layer manifest's + branch revs become the commits its comments name; prints the diff + h62.py eas-edit ENV_DIR the documented catalog edit and deviation D2, in place; prints + the diffs (nothing when the env already reads so) + h62.py uids < PODS_JSON every pod's uid, comma-separated + h62.py pick-pod OLD_UIDS < PODS_JSON " " of a Ready pod not in OLD_UIDS; exit 1 if none + h62.py model-mount CONTAINER < POD_JSON + where CONTAINER mounts the model-store PVC + h62.py verify-script SIBLINGS_JSON a self-contained script that checks a weights directory + against the Hub's file list (run in B5's pod) + h62.py trigger EAS_TAGS EAT_TAGS the re-run trigger, from two `git ls-remote --tags` outputs + h62.py pairs-left PAIRS_DIR B24 pairs still to run + h62.py vmtax H50_ANALYSIS H62_ANALYSIS + A1 on the host (H50) and in its guest (H62), and H50's B1 + on the host against B5 in its guest, described +""" +import difflib +import json +import os +import re +import sys + +PINS = { + # B5: EAS's release candidate and the component table of its release notes (read 2026-10-29). + "eas_repo": "https://github.com/intel/enterprise-ai-solutions.git", + "eas_tag": "v1.2.0-rc.1", + "eas_sha": "5c27fb39a95eecec41aa4e29a1188984e9c726f3", + "ei_branch": "release-1.1", + "ei_sha": "170eb3abb1956de0787727f21e356aef2b13ff24", + "eat_branch": "release-2.0.0", + "eat_sha": "036b66234801e9bd950afd90830821859780d668", + "catalog_name": "gemma-4-26b", + "stock_model_id": "google/gemma-4-26B-A4B-it", + # Deviation D2: the weights both arms serve (crescent amendment M1). + "qat_repo": "google/gemma-4-26B-A4B-it-qat-q4_0-unquantized", + "qat_rev": "f1e06dc520982d9b9edd76859fdb7ab209449949", + # The image enterprise-inference's catalog names for gemma-4-26b (by tag; the digest it served + # on 2026-10-29, the same image as crescent amendment E1's B1). Recorded, not forced. + "vllm_cpu_image": "vllm/vllm-openai-cpu:v0.28.0@sha256:197354e6475e638eb5ccca09beb078488046a82de0e204e7429bf1a804df9647", + # The guests (deviation D3): Ubuntu 24.04 (an OS EAS's configure supports), QEMU in a container. + "cloud_image_url": "https://cloud-images.ubuntu.com/releases/noble/release-20261017/ubuntu-24.04-server-cloudimg-amd64.img", + "cloud_image_sha256": "04e8f2461a16a44e78a370284471f29616cdd9baf1893df25807b71f985b6c94", + "cloud_image_bytes": 621284352, + "qemu_base": "ubuntu:26.04@sha256:6a06111c3b0a8600feadb270aec771dc239bf407b25625f93f44ab00e11f4d3c", + "qemu_pkg": "1:10.2.1+ds-1ubuntu3.3", + "genisoimage_pkg": "9:1.1.11-5", + # A1's guest: Docker CE from Docker's apt repository, and H50's node image. + "docker_ce": "5:29.9.2-1~ubuntu.24.04~noble", + "containerd": "2.4.2-3~ubuntu.24.04~noble", + "docker_gpg_fpr": "9DC858229FC7DD38854AE2D88D81803C0EBFCD88", + "python3_yaml": "6.0.1-2build2", # bench/apb/stage.py in VM a + "jq": "1.7.1-3ubuntu0.24.04.2", # integrations/eat/eat-check.sh in VM b (the smoke) + "node_image": "cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c", +} +QEMU_IMAGE = "h62-qemu:" + re.sub(r"[^A-Za-z0-9_.-]", "-", PINS["qemu_pkg"]) + +VCPUS = 96 +SMP = "96,sockets=1,cores=48,threads=2" # crescent's topology: 1 socket, 48 cores, 2 threads +HOST_RESERVE_GIB = 48 # MemAvailable left to the host's own tenants +ROOT_MAX_PCT = 90 +REACH = {"huggingface.co": "https://huggingface.co/api/models/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized", + "registry-1.docker.io": "https://registry-1.docker.io/v2/", + "registry.k8s.io": "https://registry.k8s.io/v2/"} +VMS = { + "a": {"name": "h62-vm-a", "host": "h62-a", "arm": "A1", "mem_gib": 96, "disk_gib": 200, + "ssh_port": 18822, "serve_port": 18831, "share": True}, + "b": {"name": "h62-vm-b", "host": "h62-b", "arm": "B5", "mem_gib": 192, "disk_gib": 400, + "ssh_port": 18823, "serve_port": 18832, "share": False}, +} +ARMS = { + # k3d-gpu's agent OVMS (agent manifest: Deployment ovms, label app=ovms, Service ovms:8000). + "A1": {"vm": "a", "ns": "k3d-gpu-agent", "selector": "app=ovms", "svc": "svc/ovms", "svc_port": 8000, + "container": "ovms", "extra": ""}, + # EAS's gemma-4-26b: a KServe LLMInferenceService; KServe v0.18.0 labels its workload pods so + # (GetWorkloadLabelSelector) and serves them behind -kserve-workload-svc:8000. + "B5": {"vm": "b", "ns": "llm-inference", + "selector": "app.kubernetes.io/part-of=llminferenceservice,app.kubernetes.io/name=gemma-4-26b," + "kserve.io/component=workload", + "svc": "svc/gemma-4-26b-kserve-workload-svc", "svc_port": 8000, "container": "main", + "extra": "llminferenceservice/gemma-4-26b"}, +} +TASKS = 24 # B24's task ids, one pair each + + +def arm_fields(arm): + """What bench/h62/arm.sh needs, in one call (it runs on every idle-arm read): VM, guest host, + serve port, namespace, selector, Service, Service port, container, then the extra object.""" + a = ARMS[arm] + v = VMS[a["vm"]] + return [a["vm"], v["host"], v["serve_port"], a["ns"], a["selector"], a["svc"], a["svc_port"], a["container"], + a["extra"]] + + +def mem_needed_kib(): + return (sum(v["mem_gib"] for v in VMS.values()) + HOST_RESERVE_GIB) * 1024 * 1024 + + +def host_facts(meminfo="/proc/meminfo", df_text=None, probe=None): + """MemAvailable (KiB), how full / is (df's percentage) and, per REACH host, whether its URL + answered over HTTPS at all (any HTTP status counts: a registry answers 401).""" + import subprocess + with open(meminfo) as f: + mem = {k: v for k, _, v in (ln.partition(":") for ln in f)} + if df_text is None: + df_text = subprocess.run(["df", "-P", "/"], capture_output=True, text=True).stdout + pct = int(df_text.splitlines()[-1].split()[4].rstrip("%")) + if probe is None: + def probe(url): + return subprocess.run(["curl", "-sS", "-o", "/dev/null", "--max-time", "20", url], + capture_output=True).returncode == 0 + return {"mem_available_kib": int(mem["MemAvailable"].split()[0]), "root_used_pct": pct, + "reach": {h: bool(probe(u)) for h, u in REACH.items()}} + + +def ir_verify(d): + """(lines, bad) of A1's IR directory against its manifest.json: every listed file's sha256 + and size, and the files the manifest does not list.""" + import hashlib + files = _load(os.path.join(d, "manifest.json"))["files"] + out, bad = [], 0 + for n, want in sorted(files.items()): + p = os.path.join(d, n) + h = hashlib.sha256() + with open(p, "rb") as f: + for chunk in iter(lambda: f.read(1 << 24), b""): + h.update(chunk) + ok = h.hexdigest() == want["sha256"] and os.path.getsize(p) == want["bytes"] + bad += not ok + out.append(f"{'ok ' if ok else 'BAD'} {n} {h.hexdigest()} {want['bytes']}") + extra = sorted(set(os.listdir(d)) - set(files) - {"manifest.json"}) + out.append(f"dir {d}; {len(files)} files, {bad} bad; not in the manifest: {extra}") + return out, bad + + +def ssh_config(state): + out = [] + for v in VMS.values(): + out += [f"Host {v['host']}", " HostName 127.0.0.1", f" Port {v['ssh_port']}", " User ubuntu", + f" IdentityFile {state}/id_ed25519", " IdentitiesOnly yes", f" UserKnownHostsFile {state}/known_hosts", + " StrictHostKeyChecking accept-new", " BatchMode yes", " ControlMaster auto", + f" ControlPath {state}/cm-%C", " ControlPersist 600", " ServerAliveInterval 30", ""] + return "\n".join(out) + + +def _key(pubkey): + k = pubkey.strip() + if not re.fullmatch(r"ssh-(ed25519|rsa) [A-Za-z0-9+/=]+( [^\n]*)?", k): + raise ValueError("not one ssh public key line") + return k + + +def user_data(vm, pubkey): + """cloud-init for one guest: the harness's key on the default user, no password login, and + the image's package state kept (deviation D3): no upgrade at boot and no apt timer running + alongside the installs.""" + return "\n".join([ + "#cloud-config", + f"hostname: {VMS[vm]['host']}", + "ssh_pwauth: false", + "ssh_authorized_keys:", + f" - {_key(pubkey)}", + "package_update: false", + "package_upgrade: false", + "runcmd:", + " - [systemctl, disable, --now, unattended-upgrades.service, apt-daily.timer, apt-daily-upgrade.timer]", + ""]) + + +def meta_data(vm): + h = VMS[vm]["host"] + return f"instance-id: {h}\nlocal-hostname: {h}\n" + + +def qemu_argv(vm, state, image, share=None, uid=None, gid=None, kvm_gid=None): + """The guest's QEMU, in the pinned container: KVM, -cpu host (AMX passes through), the host's + topology, user-mode networking with the ssh and serve ports on 127.0.0.1 only, a qcow2 overlay + on the cloud image, and for VM a A1's IR shared read-only.""" + v = VMS[vm] + if v["share"] and not share: + raise ValueError(f"VM {vm} needs A1's IR directory to share") + if share and not v["share"]: + raise ValueError(f"VM {vm} shares nothing") + if None in (uid, gid, kvm_gid): + raise ValueError("uid, gid and the kvm group are required") + argv = ["docker", "run", "-d", "--name", v["name"], "--network", "host", "--device", "/dev/kvm", + "--user", f"{uid}:{gid}", "--group-add", str(kvm_gid), "--label", "io.k3d-gpu.h62=vm", + "-v", f"{state}:/h62"] + if share: + argv += ["-v", f"{share}:/share:ro"] + fwd = f"hostfwd=tcp:127.0.0.1:{v['ssh_port']}-:22,hostfwd=tcp:127.0.0.1:{v['serve_port']}-:{v['serve_port']}" + argv += [image, "qemu-system-x86_64", "-name", v["name"], "-machine", "q35,accel=kvm", "-cpu", "host", + "-smp", SMP, "-m", f"{v['mem_gib']}G", + "-drive", f"file=/h62/{vm}/disk.qcow2,if=virtio,format=qcow2", + "-drive", f"file=/h62/{vm}/seed.iso,if=virtio,format=raw,readonly=on", + "-netdev", f"user,id=n0,{fwd}", "-device", "virtio-net-pci,netdev=n0"] + if share: + argv += ["-virtfs", "local,path=/share,mount_tag=h62share,security_model=none,readonly=on"] + return argv + ["-display", "none", "-serial", "stdio", "-monitor", "none"] + + +def gates(phase, f): + """[(ok, message)] of one gate phase, from its facts.""" + out = [] + if phase == "host": + need = mem_needed_kib() + have = f.get("mem_available_kib") + out.append((have is not None and have >= need, + f"MemAvailable {have} KiB >= {need} KiB (both guests + {HOST_RESERVE_GIB} GiB)")) + pct = f.get("root_used_pct") + out.append((pct is not None and pct <= ROOT_MAX_PCT, f"/ used {pct}% <= {ROOT_MAX_PCT}%")) + reach = f.get("reach") or {} + for h in REACH: + out.append((reach.get(h) is True, f"{h} reachable")) + elif phase == "guest": + for vm in VMS: + amx = (f.get("amx_tile") or {}).get(vm) + out.append((amx == VCPUS, f"VM {vm}: amx_tile on {amx} of {VCPUS} vCPUs")) + n = (f.get("nproc") or {}).get(vm) + out.append((n == VCPUS, f"VM {vm}: nproc {n} == {VCPUS}")) + elif phase == "nfd": + out.append((f.get("amxbf16") == "true", f"VM b node label cpu-cpuid.AMXBF16={f.get('amxbf16')}")) + elif phase == "weights": + ok = f.get("bad") == 0 and f.get("missing") == 0 and (f.get("checked") or 0) > 0 + out.append((ok, f"B5's weights: {f.get('checked')} LFS files checked, {f.get('bad')} differ, " + f"{f.get('missing')} missing, against {PINS['qat_repo']}@{PINS['qat_rev']}")) + else: + raise ValueError(f"no gate phase {phase}") + return out + + +def _diff(old, new, name): + return "".join(difflib.unified_diff(old.splitlines(True), new.splitlines(True), name, name)) + + +def eas_pin(text): + """Deviation D1: in EAS's agent-toolkit layer manifest, each layer's branch rev becomes the + commit Intel's own comment on that line names (the rc.1 release notes' component table). + Refuses anything else: another branch, a comment naming another commit, a missing layer.""" + want = {"inference": (PINS["ei_branch"], PINS["ei_sha"]), "agent-toolkit": (PINS["eat_branch"], PINS["eat_sha"])} + lines = text.splitlines(True) + starts = [i for i, ln in enumerate(lines) if re.match(r'\s*- layer: "', ln)] + seen = set() + for k, i in enumerate(starts): + layer = re.match(r'\s*- layer: "([^"]+)"', lines[i]).group(1) + if layer not in want: + continue + seen.add(layer) + branch, sha = want[layer] + end = starts[k + 1] if k + 1 < len(starts) else len(lines) + revs = [j for j in range(i, end) if re.match(r"\s*rev:", lines[j])] + if len(revs) != 1: + raise ValueError(f"layer {layer}: {len(revs)} rev lines, want 1") + j = revs[0] + indent = re.match(r"(\s*)", lines[j]).group(1) + pinned = f'{indent}rev: "{sha}" # H62 D1, was "{branch}"\n' + if lines[j] == pinned: + continue + m = re.fullmatch(r'\s*rev: "([^"]+)" # pin ([0-9a-f]{40}) .*\n?', lines[j]) + if not m or m.group(1) != branch or m.group(2) != sha: + raise ValueError(f"layer {layer}: rev line is not {branch!r} pinned to {sha}: {lines[j].strip()}") + lines[j] = pinned + if seen != set(want): + raise ValueError(f"layers missing from the manifest: {sorted(set(want) - seen)}") + return "".join(lines) + + +def set_autodeploy(text, name): + """The documented edit (single-node-deployment.md step 3): the one live + llm_services_autodeploy_models line names one catalog value.""" + pat = re.compile(r"^llm_services_autodeploy_models:.*$", re.M) + if len(pat.findall(text)) != 1: + raise ValueError("want exactly one live llm_services_autodeploy_models line") + return pat.sub(f'llm_services_autodeploy_models: ["{name}"]', text) + + +def set_model_id(text, name, old_id, new_id): + """Deviation D2: the model_id of catalog entry `name` (and only that entry) moves from the + stock id to new_id. Refuses when the entry or its stock id is not there.""" + lines = text.splitlines(True) + heads = [i for i, ln in enumerate(lines) if re.match(r"\s*- name:\s*\S", ln)] + for k, i in enumerate(heads): + if re.match(r"\s*- name:\s*(\S+)\s*$", lines[i]).group(1) != name: + continue + end = heads[k + 1] if k + 1 < len(heads) else len(lines) + ids = [j for j in range(i, end) if re.match(r"\s*model_id:", lines[j])] + if len(ids) != 1: + raise ValueError(f"{name}: {len(ids)} model_id lines") + j = ids[0] + indent, value = re.match(r"(\s*)model_id:\s*(\S+)", lines[j]).groups() + if value == new_id: + return text + if value != old_id: + raise ValueError(f"{name}: model_id is {value}, not the stock {old_id}") + lines[j] = f"{indent}model_id: {new_id}\n" + return "".join(lines) + raise ValueError(f"no catalog entry {name}") + + +def eas_edit(env_dir): + """Both edits on an env directory; returns their diffs ("" when nothing changed).""" + out = [] + for name, fn in (("config.agent-toolkit.yaml", lambda t: set_autodeploy(t, PINS["catalog_name"])), + ("models.yaml", lambda t: set_model_id(t, PINS["catalog_name"], PINS["stock_model_id"], + PINS["qat_repo"]))): + p = os.path.join(env_dir, name) + with open(p) as f: + old = f.read() + new = fn(old) + if new != old: + with open(p, "w") as f: + f.write(new) + out.append(_diff(old, new, name)) + return "".join(out) + + +def _ready(item): + st = item.get("status") or {} + return (st.get("phase") == "Running" and not (item.get("metadata") or {}).get("deletionTimestamp") + and any(c.get("type") == "Ready" and c.get("status") == "True" for c in st.get("conditions") or [])) + + +def pick_pod(doc, old): + """(name, uid) of the newest Running, Ready, not-deleting pod whose uid is not in old.""" + new = [i for i in doc.get("items", []) if i["metadata"]["uid"] not in old and _ready(i)] + if not new: + return None + m = max(new, key=lambda i: i["metadata"].get("creationTimestamp", ""))["metadata"] + return m["name"], m["uid"] + + +def model_mount(pod, container, claim="model-store"): + """Where `container` mounts the PVC `claim` (EAS's model store).""" + vols = {v["name"] for v in pod["spec"].get("volumes", []) + if (v.get("persistentVolumeClaim") or {}).get("claimName") == claim} + for c in pod["spec"].get("containers", []): + if c["name"] == container: + for m in c.get("volumeMounts", []): + if m["name"] in vols: + return m["mountPath"] + raise ValueError(f"container {container} mounts no volume of PVC {claim}") + + +VERIFY = r'''#!/usr/bin/env python3 +# Generated by bench/h62/h62.py verify-script: DIR against the Hub's file list of +# @REPO@@@REV@ (H62, B5's weights). Every LFS file's sha256, every other listed file's size, and +# the commit `hf download --local-dir` recorded per file. Last line: a JSON summary. +import glob, hashlib, json, os, sys +SIB = json.loads(@SIB@) +d = sys.argv[1] +bad = missing = checked = 0 +for s in SIB: + p = os.path.join(d, s["rfilename"]) + lfs = s.get("lfs") + if not os.path.exists(p): + print(("MISSING " if lfs else "absent ") + s["rfilename"]) + missing += bool(lfs) + continue + if lfs: + h = hashlib.sha256() + with open(p, "rb") as f: + for chunk in iter(lambda: f.read(1 << 24), b""): + h.update(chunk) + ok = h.hexdigest() == lfs["sha256"] + checked += 1 + print(("ok " if ok else "BAD ") + h.hexdigest() + " " + s["rfilename"]) + else: + ok = os.path.getsize(p) == s.get("size", os.path.getsize(p)) + print(("ok " if ok else "BAD ") + "size " + str(os.path.getsize(p)) + " " + s["rfilename"]) + bad += not ok +revs = set() +for m in glob.glob(os.path.join(d, ".cache", "huggingface", "download", "**", "*.metadata"), recursive=True): + with open(m) as f: + revs.add(f.readline().strip()) +print(json.dumps({"want": "@REV@", "revisions": sorted(revs), "checked": checked, "bad": bad, "missing": missing})) +sys.exit(1 if bad or missing else 0) +''' + + +def verify_script(siblings): + """A stdlib script that checks a weights directory against `siblings` (the Hub API's + /api/models//revision/?blobs=true answer); it needs no network.""" + sib = [{k: s[k] for k in ("rfilename", "size", "lfs") if k in s} for s in siblings["siblings"]] + return (VERIFY.replace("@SIB@", repr(json.dumps(sib))).replace("@REPO@", PINS["qat_repo"]) + .replace("@REV@", PINS["qat_rev"])) + + +def _tags(ls_remote): + return {m.group(1) for m in re.finditer(r"refs/tags/([^\s^]+)(?:\^\{\})?\s*$", ls_remote, re.M)} + + +def _release(tag, pattern, at_least): + m = re.fullmatch(pattern, tag) + return bool(m) and tuple(int(x) for x in m.groups()) >= at_least + + +def trigger(eas_ls_remote, eat_ls_remote): + """The re-run trigger: an EAS release tag at v1.2.0 or later (vX.Y.Z, not a release + candidate), or an EAT release tag at 2.0.0 or later (EAT tags its releases release-X.Y.Z and + its hotfixes release-X.Y.Z-hotfix; a vX.Y.Z tag counts too). Full support may ship under a + later number than v1.2.x or 2.x, so any later release fires.""" + eas = sorted(t for t in _tags(eas_ls_remote) if _release(t, r"v(\d+)\.(\d+)\.(\d+)", (1, 2, 0))) + eat = sorted(t for t in _tags(eat_ls_remote) + if _release(t, r"(?:release-|v)?(\d+)\.(\d+)\.(\d+)(?:-hotfix)?", (2, 0, 0))) + return {"eas": eas, "eat": eat} + + +def pairs_left(pairs_dir, total=TASKS): + """B24 pairs not yet recorded (complete or void): pairs resume across holds (pairs.next_pair).""" + sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "eatflow")) + import pairs # noqa: E402 (bench/eatflow/pairs.py) + return max(0, total - pairs.next_pair(pairs_dir)) + + +def vmtax(h50, h62): + """Described, never gated. A1 on the host (H50's analysis) against A1 in its guest (H62's): + the same stack, so the guest alone. And H50's B1 on the host against B5 in its guest: the same + vLLM CPU v0.28.0 image and QAT weights under another deployment, so the guest and the + deployment together; it shows whether the guest costs vLLM's arm more than OVMS's.""" + def arm(a, name): + s = (a.get("within_budget_success") or {}).get(name) + if not s: + return None + return {"success": f"{s['k']}/{s['n']}", "attempt_median_s": (a.get("attempt_median_s") or {}).get(name)} + + def ratio(host, guest): + if host and guest and host["attempt_median_s"] and guest["attempt_median_s"]: + return round(guest["attempt_median_s"] / host["attempt_median_s"], 3) + return None + a_host, a_guest, b_host, b_guest = arm(h50, "A1"), arm(h62, "A1"), arm(h50, "B1"), arm(h62, "B5") + return {"a1_host": a_host, "a1_guest": a_guest, "attempt_median_guest_over_host": ratio(a_host, a_guest), + "b1_host": b_host, "b5_guest": b_guest, "b_attempt_median_guest_over_host": ratio(b_host, b_guest), + "b_note": "B1 (EAT release-1.1.0's chart) and B5 (EAS's KServe deployment) share the vLLM image and " + "weights, not the deployment: the guest and the deployment are read together", + "gated": False} + + +def _load(path): + with open(path) as f: + return json.load(f) + + +def main(argv): + if not argv: + print(__doc__) + return 2 + cmd, args = argv[0], argv[1:] + if cmd == "pin": + if args[0] not in PINS: + print(f"h62.py: no pin {args[0]}", file=sys.stderr) + return 2 + print(PINS[args[0]]) + elif cmd == "qemu-image": + print(QEMU_IMAGE) + elif cmd == "vm": + if args[0] not in VMS or args[1] not in VMS[args[0]]: + print(f"h62.py: no VM field {args}", file=sys.stderr) + return 2 + print(VMS[args[0]][args[1]]) + elif cmd == "arm-fields": + print(*arm_fields(args[0])) + elif cmd == "arm": + if args[0] not in ARMS or args[1] not in ARMS[args[0]]: + print(f"h62.py: no arm field {args}", file=sys.stderr) + return 2 + print(ARMS[args[0]][args[1]]) + elif cmd == "ssh-config": + sys.stdout.write(ssh_config(args[0])) + elif cmd == "user-data": + with open(args[1]) as f: + sys.stdout.write(user_data(args[0], f.read())) + elif cmd == "meta-data": + sys.stdout.write(meta_data(args[0])) + elif cmd == "qemu-argv": + vm, state, image, uid, gid, kvm = args[:6] + print("\n".join(qemu_argv(vm, state, image, share=args[6] if len(args) > 6 else None, + uid=int(uid), gid=int(gid), kvm_gid=int(kvm)))) + elif cmd == "host-facts": + print(json.dumps(host_facts())) + elif cmd == "ir-verify": + lines, bad = ir_verify(args[0]) + print("\n".join(lines)) + return 1 if bad else 0 + elif cmd == "gates": + res = gates(args[0], _load(args[1])) + for ok, msg in res: + print(("PASS " if ok else "FAIL ") + msg) + return 0 if all(ok for ok, _ in res) else 1 + elif cmd == "eas-pin": + with open(args[0]) as f: + old = f.read() + new = eas_pin(old) + if new != old: + with open(args[0], "w") as f: + f.write(new) + sys.stdout.write(_diff(old, new, os.path.basename(args[0]))) + elif cmd == "eas-edit": + sys.stdout.write(eas_edit(args[0])) + elif cmd == "uids": + print(",".join(i["metadata"]["uid"] for i in json.load(sys.stdin).get("items", []))) + elif cmd == "pick-pod": + p = pick_pod(json.load(sys.stdin), {u for u in (args[0] if args else "").split(",") if u}) + if not p: + return 1 + print(*p) + elif cmd == "model-mount": + print(model_mount(json.load(sys.stdin), args[0])) + elif cmd == "verify-script": + sys.stdout.write(verify_script(_load(args[0]))) + elif cmd == "trigger": + with open(args[0]) as f, open(args[1]) as g: + t = trigger(f.read(), g.read()) + fired = [f"{k} {v}" for k in ("eas", "eat") for v in t[k]] + print("fired: " + (", ".join(fired) if fired else "none")) + elif cmd == "pairs-left": + print(pairs_left(args[0])) + elif cmd == "vmtax": + print(json.dumps(vmtax(_load(args[0]), _load(args[1])), indent=1)) + else: + print(__doc__) + return 2 + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main(sys.argv[1:])) + except (ValueError, KeyError, IndexError, OSError) as e: + print(f"h62.py: {e}", file=sys.stderr) + sys.exit(2) diff --git a/bench/h62/pairs-hold.sh b/bench/h62/pairs-hold.sh new file mode 100755 index 00000000..1bd0d252 --- /dev/null +++ b/bench/h62/pairs-hold.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# pairs-hold.sh N [smoke]: one H62 hold on crescent (EXPERIMENTS.md "H62 pre-registration") under +# bench/eatflow/crescent/hold.sh: the lock, the hold-start rule, the 97% disk guard, the serving +# sampler and the cleanup, LIVE_CAP 2370 inside a 2400 s timeout. live.sh runs H50's hold with both +# servers in their KVM guests (arm-A1, arm-B5): each hold restarts both serving pods first. +# Records: ~/k3dgpu-bench/results/H62/hold, pairs in ~/k3dgpu-bench/results/H62/pairs, the +# hold's own records in ~/k3dgpu-bench/runs/H62-hold. +# smoke: the wiring smoke instead (H62-smoke, the pair-start guard past the hold so no pair starts, +# records under ~/k3dgpu-bench/smoke/H62); it is not one of the 8 holds. +set -uo pipefail +N=${1:?usage: pairs-hold.sh N [smoke]} +B=${HOME}/k3dgpu-bench +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +case ${2:-hold} in + hold) name=H62-hold${N} R=${B}/results/H62 guard=600 ;; + smoke) name=H62-smoke R=${B}/smoke/H62 guard=99999 ;; + *) echo "usage: pairs-hold.sh N [smoke]"; exit 2 ;; +esac +mkdir -p "${R}/pairs" +# The guests are this campaign's arms: the serving covariate records them as vm_cpu (serving.py). +export SERVING_VMS=h62-vm-a,h62-vm-b +exec "${HERE}/../eatflow/crescent/hold.sh" "${name}" 2400 env \ + LIVE_MODE=pairs LIVE_CAP=2370 LIVE_HOLD="${N}" LIVE_PAIRS_DIR="${R}/pairs" LIVE_PAIRS_GUARD="${guard}" \ + LIVE_MODEL_SET=gemma4 LIVE_MODELS="${B}/models" LIVE_BLUEPRINT="${B}/eab" LIVE_PY="${B}/mafvenv/bin/python" \ + LIVE_REMOTE_A="${HERE}/arm-A1" LIVE_REMOTE_B="${HERE}/arm-B5" H62_STATE="${H62_STATE:-${B}/h62}" \ + PATH="${B}/bin:${PATH}" \ + bash "${HERE}/../eatflow/live.sh" H62 "${R}/hold${N}" diff --git a/bench/h62/qemu.Dockerfile b/bench/h62/qemu.Dockerfile new file mode 100644 index 00000000..e65ead46 --- /dev/null +++ b/bench/h62/qemu.Dockerfile @@ -0,0 +1,12 @@ +# The QEMU that runs H62's two KVM guests on crescent (EXPERIMENTS.md "H62 pre-registration", +# deviation D3), in a container so the host gets no package: Ubuntu 26.04's qemu-system-x86 (QEMU +# 10.2, which passes AMX through to a guest from QEMU 7.0 on), qemu-img and genisoimage for the +# cloud-init seed. Every version is a pin of bench/h62/h62.py (PINS); bench/h62/vm.sh builds it. +FROM ubuntu:26.04@sha256:6a06111c3b0a8600feadb270aec771dc239bf407b25625f93f44ab00e11f4d3c +RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + qemu-system-x86=1:10.2.1+ds-1ubuntu3.3 \ + qemu-utils=1:10.2.1+ds-1ubuntu3.3 \ + genisoimage=9:1.1.11-5 \ + && dpkg-query -W > /h62-packages.txt \ + && rm -rf /var/lib/apt/lists/* diff --git a/bench/h62/setup.sh b/bench/h62/setup.sh new file mode 100755 index 00000000..fdb27e40 --- /dev/null +++ b/bench/h62/setup.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# setup.sh: H62's setup phase on crescent (EXPERIMENTS.md "H62 pre-registration", "Setup"), not a +# hold, inside the bench lock. Run from the H62 checkout at the deploy commit. +# 1. host gate: MemAvailable for both guests plus 48 GiB, / at most 90% used, Hugging Face and the +# registries reachable; re-read every 10 min for up to 60 min +# 2. A1's IR against its manifest.json; the Hub's file list of B5's weights (the QAT checkpoint +# at f1e06dc5) and the script that checks them +# 3. both guests (vm.sh build, fetch, up, wait), then the guest gate: amx_tile on all 96 vCPUs +# 4. VM a's A1 (guest-a.sh, 45 min ceiling) and VM b's EAS install (guest-b.sh, 120 min ceiling), +# side by side; their records copied back +# 5. the NFD gate (VM b's node carries cpu-cpuid.AMXBF16=true) and the weight gate (B5's files +# equal the QAT checkpoint's) +# Exit 0: both arms serve. 4: lock not taken. 6: the host gate still failed after 60 min. 7: not +# deliverable (a ceiling passed, or no AMX in a guest). 8: the NFD or weight gate failed (no hold +# until a dated amendment). 1: anything else. Records: ~/k3dgpu-bench/results/H62/setup. +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +P() { python3 "${HERE}/h62.py" "$@"; } +B=${HOME}/k3dgpu-bench +export H62_STATE=${H62_STATE:-${B}/h62} +R=${H62_RESULTS:-${B}/results/H62}/setup +A_CAP=2700 B_CAP=7200 +mkdir -p "${R}" +log() { echo "== setup $* $(date -u +%FT%TZ)" | tee -a "${R}/setup.log"; } +# Every ssh, scp and vm.sh below runs with fd 9 closed: an ssh ControlMaster it starts outlives +# this script, and must not keep the bench lock. +G() { ssh -F "${H62_STATE}/ssh_config" "$(P vm "$1" host)" "${@:2}" 9>&-; } +put() { scp -q -F "${H62_STATE}/ssh_config" "${@:2}" "$(P vm "$1" host):h62/" 9>&-; } +vm() { bash "${HERE}/vm.sh" "$@" 9>&-; } +exec 9> "${HOME}/k3dgpu-bench.lock" +flock -w 7200 9 || { log "lock not taken in 7200 s"; exit 4; } +log "lock taken" +git -C "${REPO}" rev-parse HEAD > "${R}/commit.txt" +git -C "${REPO}" status --porcelain > "${R}/commit-status.txt" + +# 1. Host gate. +for i in 0 1 2 3 4 5 6; do + P host-facts > "${R}/host-facts.json" + P gates host "${R}/host-facts.json" > "${R}/gate-host.txt" && break + [ "${i}" -lt 6 ] || { log "host gate failed after 60 min"; cat "${R}/gate-host.txt"; exit 6; } + log "host gate not passed: $(grep -c FAIL "${R}/gate-host.txt") item(s); re-read in 10 min" + sleep 600 +done +log "host gate passed" + +# 2. A1's IR and B5's reference list. +IR=$(readlink -f "${H62_IR:-${B}/models/gemma-4-26B-A4B-it-qat-int4-ov}") +P ir-verify "${IR}" > "${R}/ir-verify.txt" || { log "A1's IR does not match its manifest"; exit 1; } +curl -fsSL "https://huggingface.co/api/models/$(P pin qat_repo)/revision/$(P pin qat_rev)?blobs=true" \ + > "${R}/qat-siblings.json" || { log "no Hub file list"; exit 1; } +P verify-script "${R}/qat-siblings.json" > "${R}/verify-weights.py" + +# 3. The guests. +log "guests" +vm build > "${R}/vm-build.log" 2>&1 || { log "QEMU image build failed"; exit 1; } +vm fetch > "${R}/vm-fetch.log" 2>&1 || { log "cloud image failed its check"; exit 1; } +for vm in a b; do + H62_IR=${IR} vm up "${vm}" > "${R}/vm-up-${vm}.log" 2>&1 || { log "VM ${vm} did not start"; exit 1; } +done +for vm in a b; do + vm wait "${vm}" 900 >> "${R}/vm-up-${vm}.log" 2>&1 || { log "VM ${vm} did not boot"; exit 1; } +done +fa=$(vm facts a) && fb=$(vm facts b) || { log "no guest facts"; exit 1; } +python3 -c 'import json, sys; a, b = (json.loads(x) for x in sys.argv[1:3]) +print(json.dumps({"amx_tile": {"a": a["amx_tile"], "b": b["amx_tile"]}, "nproc": {"a": a["nproc"], "b": b["nproc"]}, + "a": a, "b": b}))' "${fa}" "${fb}" > "${R}/guest-facts.json" +P gates guest "${R}/guest-facts.json" > "${R}/gate-guest.txt" || { log "guest gate failed: not deliverable"; exit 7; } + +# 4. Each guest's setup, side by side under its ceiling. +for vm in a b; do + G "${vm}" mkdir -p h62/bin + put "${vm}" "${HERE}/h62.py" "${HERE}/guest-${vm}.sh" +done +scp -q -F "${H62_STATE}/ssh_config" "${B}/bin/k3d" "${B}/bin/kubectl" h62-a:h62/bin/ 9>&- +git -C "${REPO}" archive --format=tar HEAD | G a "mkdir -p h62/k3d-gpu && tar -x -C h62/k3d-gpu" +log "guest-a (cap ${A_CAP} s) and guest-b (cap ${B_CAP} s)" +timeout -k 60 "${A_CAP}" ssh -F "${H62_STATE}/ssh_config" h62-a 'bash h62/guest-a.sh' > "${R}/guest-a.log" 2>&1 9>&- & +pa=$! +timeout -k 60 "${B_CAP}" ssh -F "${H62_STATE}/ssh_config" h62-b 'bash h62/guest-b.sh' > "${R}/guest-b.log" 2>&1 9>&- & +pb=$! +wait "${pa}"; xa=$? +wait "${pb}"; xb=$? +for vm in a b; do + scp -q -r -F "${H62_STATE}/ssh_config" "$(P vm "${vm}" host):h62/records" "${R}/guest-${vm}" 2>/dev/null 9>&- || true +done +log "guest-a exit ${xa}, guest-b exit ${xb}" +if [ "${xa}" = 124 ] || [ "${xb}" = 124 ]; then log "a ceiling passed: not deliverable"; exit 7; fi +[ "${xa}" = 0 ] && [ "${xb}" = 0 ] || exit 1 + +# 5. NFD and weight gates on VM b. +amx=$(G b ". ~/h62/env && kubectl get nodes -o jsonpath='{.items[0].metadata.labels.feature\\.node\\.kubernetes\\.io/cpu-cpuid\\.AMXBF16}'") +python3 -c 'import json, sys; print(json.dumps({"amxbf16": sys.argv[1] or None}))' "${amx}" > "${R}/nfd-facts.json" +P gates nfd "${R}/nfd-facts.json" > "${R}/gate-nfd.txt" || { log "NFD gate failed"; exit 8; } +sel=$(P arm B5 selector) +read -r pod _ <<< "$(G b ". ~/h62/env && kubectl -n llm-inference get pods -l ${sel} -o json" | P pick-pod "")" +[ -n "${pod:-}" ] || { log "no Ready B5 pod"; exit 1; } +mnt=$(G b ". ~/h62/env && kubectl -n llm-inference get pod ${pod} -o json" | P model-mount main) || { log "no model mount"; exit 1; } +G b ". ~/h62/env && kubectl -n llm-inference exec -i ${pod} -c main -- python3 - ${mnt}" \ + < "${R}/verify-weights.py" > "${R}/weights-verify.txt" 2>&1 +tail -1 "${R}/weights-verify.txt" > "${R}/weights-facts.json" +P gates weights "${R}/weights-facts.json" > "${R}/gate-weights.txt" || { log "weight gate failed"; exit 8; } +log "setup done: both arms serve" diff --git a/bench/h62/smoke.sh b/bench/h62/smoke.sh new file mode 100755 index 00000000..7b5d3024 --- /dev/null +++ b/bench/h62/smoke.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# smoke.sh: H62's wiring smoke (EXPERIMENTS.md "H62 pre-registration", "Wiring smoke"), after +# setup.sh and before hold 1. It scores nothing, is not one of the 8 holds, and is disclosed in +# the run section. +# 1. B5 through EAS's own stack, in VM b, inside the bench lock: `es_auto_installer.sh validate +# agent-toolkit`, then integrations/eat/eat-check.sh in EAS mode (PR #260's interface) through +# EAS's gateway on gemma-4-26b: a chat, a get_weather tool call, and a sandbox run that +# prints 385 +# 2. one H62 hold with the pair-start guard past its end (pairs-hold.sh 0 smoke): both serving +# pods restart and answer through the harness's LiteLLM, both warm-ups answer, no pair starts +# Exit 0 when both pass; 1 otherwise; 4 when the lock is not taken. Records: ~/k3dgpu-bench/smoke/H62. +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +P() { python3 "${HERE}/h62.py" "$@"; } +B=${HOME}/k3dgpu-bench +export H62_STATE=${H62_STATE:-${B}/h62} +S=${B}/smoke/H62 +MODEL=$(P pin catalog_name) +CHECK=${REPO}/integrations/eat/eat-check.sh +mkdir -p "${S}" +log() { echo "== smoke $* $(date -u +%FT%TZ)" | tee -a "${S}/smoke.log"; } +G() { ssh -F "${H62_STATE}/ssh_config" h62-b "$@" 9>&-; } +grep -qE '^[[:space:]]+eas\)' "${CHECK}" || { log "eat-check.sh at this commit has no EAS mode (PR #260): no smoke"; exit 1; } +exec 9> "${HOME}/k3dgpu-bench.lock" +flock -w 7200 9 || { log "lock not taken in 7200 s"; exit 4; } +log "lock taken; step 1 in VM b" +G "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q jq=$(P pin jq)" > "${S}/jq.log" 2>&1 \ + || { log "jq not installed in VM b"; exit 1; } +scp -q -F "${H62_STATE}/ssh_config" "${CHECK}" h62-b:h62/eat-check.sh 9>&- +G "cd ~/enterprise-ai-solutions && ./es_auto_installer.sh validate agent-toolkit" > "${S}/validate.log" 2>&1 +xv=$? +G ". ~/h62/env && EAT_MODE=eas EAT_MODEL=${MODEL} bash ~/h62/eat-check.sh" > "${S}/eat-check.log" 2>&1 +xc=$? +log "validate exit ${xv}, eat-check exit ${xc}" +exec 9>&- +log "step 2: H62-smoke" +bash "${HERE}/pairs-hold.sh" 0 smoke > "${S}/hold.log" 2>&1 +xh=$? +warm=$(python3 -c 'import json, sys +rows = [json.loads(x) for x in open(sys.argv[1]) if x.strip()] +print(sum(1 for r in rows if not r.get("error")))' "${S}/pairs/warmups.jsonl" 2>/dev/null || echo 0) +pairs=$(grep -c . "${S}/pairs/pairs.jsonl" 2>/dev/null || true) +log "hold exit ${xh}, warm-ups answered ${warm} of 2, pairs ${pairs:-0}" +[ "${xv}" = 0 ] && [ "${xc}" = 0 ] && [ "${xh}" = 0 ] && [ "${warm}" = 2 ] && [ "${pairs:-0}" = 0 ] || exit 1 +log "wiring smoke passed" diff --git a/bench/h62/teardown.sh b/bench/h62/teardown.sh new file mode 100755 index 00000000..3ca482e7 --- /dev/null +++ b/bench/h62/teardown.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# teardown.sh: H62's teardown (EXPERIMENTS.md "H62 pre-registration"), after the last hold or any +# abort. Both guests are powered off and removed (vm.sh down), and their disks, the cloud image and +# the QEMU image are deleted (vm.sh purge); ~/k3dgpu-bench/results/H62 stays. Serving pairs that +# campaign.sh stopped are started again (gpu-serving.sh start) and then checked through Dynamo +# :8100 as crescent amendment E2 item 7 does: a 16-token chat answers with content and that pair's +# SGLang num_requests_total rises, polled for up to 180 s. The result is recorded; it gates nothing. +set -uo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "${HERE}/../.." && pwd) +B=${HOME}/k3dgpu-bench +R=${B}/results/H62 +export H62_STATE=${H62_STATE:-${B}/h62} +mkdir -p "${R}" +for vm in a b; do bash "${HERE}/vm.sh" down "${vm}"; done +bash "${HERE}/vm.sh" purge +if [ -s "${R}/serving-stopped.txt" ]; then + while read -r c; do + bash "${REPO}/bench/eatflow/crescent/gpu-serving.sh" start "${c}" >> "${R}/serving.log" 2>&1 + done < "${R}/serving-stopped.txt" + mv "${R}/serving-stopped.txt" "${R}/serving-restored.txt" + n() { curl -s -m 8 "http://127.0.0.1:$1/metrics" | awk '/^sglang:num_requests_total/{s+=$2} END{print s+0}'; } + for port in 30000 30001; do + before=$(n "${port}") end=$(( SECONDS + 180 )) ok=0 c="" + until [ "${SECONDS}" -ge "${end}" ]; do + c=$(curl -s -m 60 http://127.0.0.1:8100/v1/chat/completions -H 'Content-Type: application/json' \ + -d '{"model":"Qwen3.8-27B","messages":[{"role":"user","content":"Say OK"}],"max_tokens":16,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}' \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["choices"][0]["message"]["content"].strip())' 2>/dev/null) + if [ -n "${c}" ] && [ "$(n "${port}")" -gt "${before}" ]; then ok=1; break; fi + sleep 5 + done + echo "$(date -u +%FT%TZ) :${port} pass=${ok} before=${before} after=$(n "${port}") content=${c:-none}" \ + | tee -a "${R}/frontdoor-post.txt" + done +fi diff --git a/bench/h62/test_h62.py b/bench/h62/test_h62.py new file mode 100644 index 00000000..5c0f2c6c --- /dev/null +++ b/bench/h62/test_h62.py @@ -0,0 +1,770 @@ +#!/usr/bin/env python3 +"""Tests for bench/h62 (EXPERIMENTS.md "H62 pre-registration"): the pins, the two guests' +QEMU command lines and cloud-init, the setup gates, deviation D1 (the layer manifest's +commits) and the documented edits with deviation D2, the pod and cgroup reads behind the +idle-arm check, B5's weight check, the re-run trigger, and the shell scripts against stubs. +No VM, no cluster, no network.""" +import json +import os +import shutil +import stat +import subprocess +import sys +import tempfile +import textwrap +import unittest + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import h62 # noqa: E402 + +REPO = os.path.normpath(os.path.join(HERE, "..", "..")) +GIB_KIB = 1024 * 1024 + + +def read(*path): + with open(os.path.join(*path)) as f: + return f.read() + + +def run(*argv, stdin=None, env=None): + return subprocess.run([sys.executable, os.path.join(HERE, "h62.py"), *argv], input=stdin, + capture_output=True, text=True, env=env) + + +# EAS v1.2.0-rc.1 (5c27fb3) configs/repos/repos.agent-toolkit.yaml, lines 12-28, verbatim. +MANIFEST = textwrap.dedent('''\ + repos: + - layer: "inference" + url: "https://github.com/intel/enterprise-inference" + dest: "enterprise.ai-inference" + rev: "release-1.1" # pin 170eb3abb1956de0787727f21e356aef2b13ff24 (#49) for release + deployment_subdir: "" + config_dir: "" + default_flavour: "" + model_catalog: "model_manager/models.yaml" + + - layer: "agent-toolkit" + url: "https://github.com/intel/enterprise-agent-toolkit" + dest: "enterprise.agent-toolkit" + rev: "release-2.0.0" # pin 036b66234801e9bd950afd90830821859780d668 (#19) for release + deployment_subdir: "" + config_dir: "" + default_flavour: "" + ''') + +# EAT release-2.0.0 (036b662) config.yaml, lines 33-36 (the file init seeds as +# env/local/config.agent-toolkit.yaml), and the same key commented in a doc block. +CONFIG = textwrap.dedent('''\ + # Model(s) to deploy, from the catalog (env/local/models.yaml) + # llm_services_autodeploy_models: ["qwen3-coder-30b", "bge-base-en"] + llm_services_autodeploy_models: ["qwen2-5-coder-14b"] + redis_stack_enabled: true + ''') + +# enterprise-inference release-1.1 (170eb3a) model_manager/models.yaml, lines 355-366 and +# 405-416, trimmed: the gemma-4-26b entry between two others, at the file's indentation. +MODELS = textwrap.indent(textwrap.dedent('''\ + - name: gemma-4-26b-tp2 + model_id: google/gemma-4-26B-A4B-it + category: llm + + - name: gemma-4-26b + model_id: google/gemma-4-26B-A4B-it + category: llm + runtime: vllm + server_version: "0.28.0" + cpu: auto # one NUMA node per TP rank, per node + memory: 128Gi + + - name: qwen3guard + model_id: Qwen/Qwen3Guard-Gen-0.6B + category: llm + '''), " ") + + +class PinsTest(unittest.TestCase): + """Every pin the pre-registration names, in one table the scripts read.""" + + def test_pins_are_the_preregistered_values(self): + p = h62.PINS + self.assertEqual(p["eas_tag"], "v1.2.0-rc.1") + self.assertEqual(p["eas_sha"], "5c27fb39a95eecec41aa4e29a1188984e9c726f3") + self.assertEqual(p["ei_sha"], "170eb3abb1956de0787727f21e356aef2b13ff24") + self.assertEqual(p["eat_sha"], "036b66234801e9bd950afd90830821859780d668") + self.assertEqual(p["cloud_image_sha256"], "04e8f2461a16a44e78a370284471f29616cdd9baf1893df25807b71f985b6c94") + self.assertTrue(p["cloud_image_url"].endswith("/release-20261017/ubuntu-24.04-server-cloudimg-amd64.img")) + self.assertEqual(p["cloud_image_bytes"], 621284352) + self.assertEqual(p["qemu_base"], "ubuntu:26.04@sha256:6a06111c3b0a8600feadb270aec771dc239bf407b25625f93f44ab00e11f4d3c") + self.assertEqual(p["qemu_pkg"], "1:10.2.1+ds-1ubuntu3.3") + self.assertEqual(p["docker_ce"], "5:29.9.2-1~ubuntu.24.04~noble") + self.assertEqual(p["containerd"], "2.4.2-3~ubuntu.24.04~noble") + self.assertEqual(p["docker_gpg_fpr"], "9DC858229FC7DD38854AE2D88D81803C0EBFCD88") + self.assertEqual(p["node_image"], "cryptoandcoffee/k3d-gpu:0.1.61@sha256:" + "8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c") + self.assertEqual(p["qat_repo"], "google/gemma-4-26B-A4B-it-qat-q4_0-unquantized") + self.assertEqual(p["qat_rev"], "f1e06dc520982d9b9edd76859fdb7ab209449949") + self.assertEqual((p["catalog_name"], p["stock_model_id"]), ("gemma-4-26b", "google/gemma-4-26B-A4B-it")) + self.assertEqual(p["vllm_cpu_image"], "vllm/vllm-openai-cpu:v0.28.0@sha256:" + "197354e6475e638eb5ccca09beb078488046a82de0e204e7429bf1a804df9647") + self.assertEqual((p["jq"], p["python3_yaml"]), ("1.7.1-3ubuntu0.24.04.2", "6.0.1-2build2")) + self.assertEqual(h62.QEMU_IMAGE, "h62-qemu:1-10.2.1-ds-1ubuntu3.3") + + def test_cli_prints_one_pin(self): + r = run("pin", "eas_sha") + self.assertEqual((r.returncode, r.stdout), (0, "5c27fb39a95eecec41aa4e29a1188984e9c726f3\n")) + self.assertEqual(run("pin", "nope").returncode, 2) + + def test_the_qemu_image_is_built_from_the_pins(self): + df = read(HERE, "qemu.Dockerfile") + self.assertIn(f"FROM {h62.PINS['qemu_base']}\n", df) + for pkg in ("qemu-system-x86", "qemu-utils"): + self.assertIn(f"{pkg}={h62.PINS['qemu_pkg']}", df) + self.assertIn(f"genisoimage={h62.PINS['genisoimage_pkg']}", df) + + def test_the_cloud_image_is_checked_before_use(self): + vm = read(HERE, "vm.sh") + self.assertIn('P pin cloud_image_sha256', vm) + self.assertIn('sha256sum -c', vm) + + +class VmSpecTest(unittest.TestCase): + def test_two_guests_of_one_shape(self): + a, b = h62.VMS["a"], h62.VMS["b"] + self.assertEqual((a["arm"], a["mem_gib"], b["arm"], b["mem_gib"]), ("A1", 96, "B5", 192)) + self.assertEqual(h62.SMP, "96,sockets=1,cores=48,threads=2") + self.assertEqual(h62.VCPUS, 96) + # The guests carry no k3dgpu prefix: hold.sh's cleanup never removes them, and another + # row's serving covariate counts them as foreign load. + for v in (a, b): + self.assertFalse(v["name"].startswith("k3dgpu"), v["name"]) + self.assertEqual(len({a["ssh_port"], a["serve_port"], b["ssh_port"], b["serve_port"]}), 4) + + def test_memory_gate_is_both_guests_plus_48_gib(self): + self.assertEqual(h62.mem_needed_kib(), (96 + 192 + 48) * GIB_KIB) + + +class QemuArgvTest(unittest.TestCase): + def test_vm_b(self): + argv = h62.qemu_argv("b", "/st", "h62-qemu:x", uid=1000, gid=1000, kvm_gid=992) + self.assertEqual(argv[:4], ["docker", "run", "-d", "--name"]) + self.assertEqual(argv[4], "h62-vm-b") + for pair in (["--network", "host"], ["--device", "/dev/kvm"], ["--user", "1000:1000"], + ["--group-add", "992"], ["-v", "/st:/h62"], ["-cpu", "host"], ["-m", "192G"], + ["-smp", "96,sockets=1,cores=48,threads=2"], ["-machine", "q35,accel=kvm"]): + self.assertTrue(any(argv[i:i + 2] == pair for i in range(len(argv))), pair) + net = argv[argv.index("-netdev") + 1] + self.assertEqual(net, "user,id=n0,hostfwd=tcp:127.0.0.1:18823-:22,hostfwd=tcp:127.0.0.1:18832-:18832") + self.assertNotIn("0.0.0.0", " ".join(argv)) + self.assertNotIn("-virtfs", argv) + self.assertIn("file=/h62/b/disk.qcow2,if=virtio,format=qcow2", argv) + self.assertIn("file=/h62/b/seed.iso,if=virtio,format=raw,readonly=on", argv) + self.assertEqual(argv[argv.index("h62-qemu:x") + 1], "qemu-system-x86_64") + + def test_vm_a_shares_the_ir_read_only(self): + argv = h62.qemu_argv("a", "/st", "img", share="/home/u/ir", uid=1, gid=1, kvm_gid=2) + self.assertIn("/home/u/ir:/share:ro", argv) + self.assertEqual(argv[argv.index("-virtfs") + 1], + "local,path=/share,mount_tag=h62share,security_model=none,readonly=on") + self.assertEqual(argv[argv.index("-m") + 1], "96G") + with self.assertRaises(ValueError): + h62.qemu_argv("b", "/st", "img", share="/x", uid=1, gid=1, kvm_gid=2) + with self.assertRaises(ValueError): + h62.qemu_argv("a", "/st", "img", uid=1, gid=1, kvm_gid=2) # A1's IR is required + + def test_cli_one_argument_per_line(self): + r = run("qemu-argv", "b", "/st", "img", "1000", "1000", "992") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(r.stdout.splitlines(), h62.qemu_argv("b", "/st", "img", uid=1000, gid=1000, kvm_gid=992)) + + +class CloudInitTest(unittest.TestCase): + KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExample h62" + + def test_user_data(self): + u = h62.user_data("b", self.KEY) + self.assertTrue(u.startswith("#cloud-config\n")) + self.assertIn(f" - {self.KEY}\n", u) + self.assertIn("ssh_pwauth: false\n", u) + self.assertIn("package_upgrade: false\n", u) + self.assertIn("hostname: h62-b\n", u) + # D3: the guest's package state stays the image's; no apt run competes with the install. + for unit in ("unattended-upgrades.service", "apt-daily.timer", "apt-daily-upgrade.timer"): + self.assertIn(unit, u) + + def test_meta_data_and_key_checks(self): + self.assertEqual(h62.meta_data("a"), "instance-id: h62-a\nlocal-hostname: h62-a\n") + for bad in ("", "ssh-ed25519 AAAA\nrm -rf /", "not-a-key"): + with self.assertRaises(ValueError): + h62.user_data("a", bad) + + +class HostFactsTest(unittest.TestCase): + """The host gate's facts: MemAvailable, how full / is, and whether the registries answer.""" + + def test_reads_meminfo_df_and_probes(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + mi = os.path.join(tmp, "meminfo") + with open(mi, "w") as f: + f.write("MemTotal: 527700000 kB\nMemAvailable: 374341632 kB\n") + df = "Filesystem 1024-blocks Used Available Capacity Mounted on\n/dev/x 3600000000 1100000000 2400000000 31% /\n" + seen = [] + f = h62.host_facts(mi, df, lambda url: seen.append(url) or "registry.k8s.io" not in url) + self.assertEqual((f["mem_available_kib"], f["root_used_pct"]), (374341632, 31)) + self.assertEqual(f["reach"], {"huggingface.co": True, "registry-1.docker.io": True, "registry.k8s.io": False}) + self.assertEqual(len(seen), 3) + + +class IrVerifyTest(unittest.TestCase): + """A1's IR against its manifest.json (crescent amendment E3 item 4), before VM a sees it.""" + + def test_manifest_check(self): + import hashlib + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + files = {"openvino_model.bin": b"bin" * 10, "openvino_model.xml": b""} + man = {"files": {}} + for n, data in files.items(): + with open(os.path.join(tmp, n), "wb") as f: + f.write(data) + man["files"][n] = {"sha256": hashlib.sha256(data).hexdigest(), "bytes": len(data)} + with open(os.path.join(tmp, "manifest.json"), "w") as f: + json.dump(man, f) + r = run("ir-verify", tmp) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn("2 files, 0 bad", r.stdout) + with open(os.path.join(tmp, "openvino_model.xml"), "ab") as f: + f.write(b" ") + r = run("ir-verify", tmp) + self.assertEqual(r.returncode, 1) + self.assertIn("1 bad", r.stdout) + + +class SshConfigTest(unittest.TestCase): + def test_both_guests(self): + c = h62.ssh_config("/home/u/k3dgpu-bench/h62") + for host, port in (("h62-a", 18822), ("h62-b", 18823)): + blk = c[c.index(f"Host {host}\n"):] + self.assertIn(f" Port {port}\n", blk) + for line in (" HostName 127.0.0.1\n", " User ubuntu\n", " IdentityFile /home/u/k3dgpu-bench/h62/id_ed25519\n", + " UserKnownHostsFile /home/u/k3dgpu-bench/h62/known_hosts\n", " StrictHostKeyChecking accept-new\n", + " ControlMaster auto\n", " ControlPath /home/u/k3dgpu-bench/h62/cm-%C\n", " BatchMode yes\n"): + self.assertEqual(c.count(line), 2, line) + + +class GatesTest(unittest.TestCase): + HOST = {"mem_available_kib": 357 * GIB_KIB, "root_used_pct": 31, + "reach": {"huggingface.co": True, "registry-1.docker.io": True, "registry.k8s.io": True}} + + def failed(self, phase, facts): + return [m for ok, m in h62.gates(phase, facts) if not ok] + + def test_host(self): + self.assertEqual(self.failed("host", self.HOST), []) + edge = dict(self.HOST, mem_available_kib=h62.mem_needed_kib()) + self.assertEqual(self.failed("host", edge), []) + low = dict(self.HOST, mem_available_kib=h62.mem_needed_kib() - 1) + self.assertEqual(len(self.failed("host", low)), 1) + self.assertEqual(self.failed("host", dict(self.HOST, root_used_pct=90)), []) + self.assertEqual(len(self.failed("host", dict(self.HOST, root_used_pct=91))), 1) + unreach = dict(self.HOST, reach={"huggingface.co": False, "registry-1.docker.io": True, "registry.k8s.io": True}) + self.assertIn("huggingface.co", self.failed("host", unreach)[0]) + + def test_guest_amx_on_every_vcpu(self): + good = {"amx_tile": {"a": 96, "b": 96}, "nproc": {"a": 96, "b": 96}} + self.assertEqual(self.failed("guest", good), []) + self.assertEqual(len(self.failed("guest", {"amx_tile": {"a": 96, "b": 0}, "nproc": {"a": 96, "b": 96}})), 1) + self.assertEqual(len(self.failed("guest", {"amx_tile": {"a": 96}, "nproc": {"a": 96}})), 2) # b unread + + def test_nfd_and_weights(self): + self.assertEqual(self.failed("nfd", {"amxbf16": "true"}), []) + self.assertEqual(len(self.failed("nfd", {"amxbf16": None})), 1) + self.assertEqual(self.failed("weights", {"bad": 0, "missing": 0, "checked": 3}), []) + self.assertEqual(len(self.failed("weights", {"bad": 1, "missing": 0, "checked": 3})), 1) + self.assertEqual(len(self.failed("weights", {"bad": 0, "missing": 0, "checked": 0})), 1) + with self.assertRaises(ValueError): + h62.gates("other", {}) + + def test_cli(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + p = os.path.join(tmp, "f.json") + with open(p, "w") as f: + json.dump(self.HOST, f) + r = run("gates", "host", p) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn("PASS", r.stdout) + with open(p, "w") as f: + json.dump(dict(self.HOST, root_used_pct=95), f) + r = run("gates", "host", p) + self.assertEqual(r.returncode, 1) + self.assertIn("FAIL", r.stdout) + + +class EasPinTest(unittest.TestCase): + """Deviation D1: the layer manifest's branch revs become the commits Intel's own comments + and the rc.1 release notes name; nothing else in the file changes.""" + + def test_pins_both_layers_to_the_commented_commits(self): + out = h62.eas_pin(MANIFEST) + self.assertIn('rev: "170eb3abb1956de0787727f21e356aef2b13ff24" # H62 D1, was "release-1.1"', out) + self.assertIn('rev: "036b66234801e9bd950afd90830821859780d668" # H62 D1, was "release-2.0.0"', out) + diff = [(a, b) for a, b in zip(MANIFEST.splitlines(), out.splitlines()) if a != b] + self.assertEqual(len(diff), 2) + self.assertEqual(len(out.splitlines()), len(MANIFEST.splitlines())) + self.assertEqual(h62.eas_pin(out), out) # a second run changes nothing + + def test_refuses_what_it_was_not_written_for(self): + for bad in (MANIFEST.replace("pin 170eb3ab", "pin 0000000b"), # Intel's comment names another commit + MANIFEST.replace('rev: "release-1.1"', 'rev: "main"'), # another branch + MANIFEST.replace('- layer: "inference"', '- layer: "other"'), + MANIFEST + ' rev: "x"\n'): # a second rev in a block + with self.assertRaises(ValueError): + h62.eas_pin(bad) + + +class EasEditTest(unittest.TestCase): + """The one documented edit (the catalog value) and deviation D2 (gemma-4-26b's weights).""" + + def test_autodeploy_gemma_only_on_the_live_line(self): + out = h62.set_autodeploy(CONFIG, "gemma-4-26b") + self.assertIn('\nllm_services_autodeploy_models: ["gemma-4-26b"]\n', out) + self.assertIn('# llm_services_autodeploy_models: ["qwen3-coder-30b", "bge-base-en"]\n', out) + self.assertEqual(h62.set_autodeploy(out, "gemma-4-26b"), out) + with self.assertRaises(ValueError): + h62.set_autodeploy(CONFIG.replace("llm_services_autodeploy_models: [\"qwen", "#x: [\"qwen"), "gemma-4-26b") + + def test_d2_moves_only_gemma_4_26b_to_the_qat_weights(self): + out = h62.set_model_id(MODELS, "gemma-4-26b", "google/gemma-4-26B-A4B-it", + "google/gemma-4-26B-A4B-it-qat-q4_0-unquantized") + blocks = out.split("\n\n") + self.assertIn("model_id: google/gemma-4-26B-A4B-it\n", blocks[0]) # the tp2 entry keeps its id + self.assertIn("model_id: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized\n", blocks[1]) + self.assertIn("model_id: Qwen/Qwen3Guard-Gen-0.6B\n", blocks[2]) + self.assertEqual(sum(1 for a, b in zip(MODELS.splitlines(), out.splitlines()) if a != b), 1) + with self.assertRaises(ValueError): + h62.set_model_id(MODELS, "nope", "a", "b") + with self.assertRaises(ValueError): # the stock id is not there + h62.set_model_id(MODELS, "gemma-4-26b", "google/other", "b") + + def test_cli_edits_an_env_and_prints_the_diffs(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + for name, text in (("config.agent-toolkit.yaml", CONFIG), ("models.yaml", MODELS), + ("global_config.yaml", 'base_domain_name: "solutions.ai"\n')): + with open(os.path.join(tmp, name), "w") as f: + f.write(text) + r = run("eas-edit", tmp) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('+llm_services_autodeploy_models: ["gemma-4-26b"]', r.stdout) + self.assertIn("+ model_id: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized", r.stdout) + self.assertEqual(read(tmp, "global_config.yaml"), 'base_domain_name: "solutions.ai"\n') # untouched + self.assertEqual(run("eas-edit", tmp).stdout, "") # idempotent + p = os.path.join(tmp, "manifest.yaml") + with open(p, "w") as f: + f.write(MANIFEST) + r = run("eas-pin", p) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('+ rev: "170eb3abb1956de0787727f21e356aef2b13ff24"', r.stdout) + + +def pod(name, uid, ready=True, deleting=False, created="2026-10-30T00:00:00Z", phase="Running"): + meta = {"name": name, "uid": uid, "creationTimestamp": created} + if deleting: + meta["deletionTimestamp"] = "2026-10-30T00:01:00Z" + return {"metadata": meta, "status": {"phase": phase, + "conditions": [{"type": "Ready", "status": "True" if ready else "False"}]}} + + +class PodTest(unittest.TestCase): + def test_pick_a_new_ready_pod(self): + old = pod("p0", "u0", deleting=True) + self.assertIsNone(h62.pick_pod({"items": [old, pod("p1", "u1", ready=False)]}, {"u0"})) + self.assertEqual(h62.pick_pod({"items": [old, pod("p1", "u1")]}, {"u0"}), ("p1", "u1")) + self.assertIsNone(h62.pick_pod({"items": [pod("p0", "u0")]}, {"u0"})) # the old pod is not new + self.assertEqual(h62.pick_pod({"items": [pod("p0", "u0")]}, set()), ("p0", "u0")) + two = {"items": [pod("p1", "u1", created="2026-10-30T00:00:01Z"), pod("p2", "u2", created="2026-10-30T00:00:09Z")]} + self.assertEqual(h62.pick_pod(two, set()), ("p2", "u2")) + self.assertIsNone(h62.pick_pod({"items": [pod("p1", "u1", phase="Pending")]}, set())) + + def test_cli(self): + doc = json.dumps({"items": [pod("p0", "u0", deleting=True), pod("p1", "u1")]}) + self.assertEqual(run("uids", stdin=doc).stdout, "u0,u1\n") + r = run("pick-pod", "u0", stdin=doc) + self.assertEqual((r.returncode, r.stdout), (0, "p1 u1\n")) + self.assertEqual(run("pick-pod", "u0,u1", stdin=doc).returncode, 1) + self.assertEqual(run("pick-pod", "", stdin=doc).stdout, "p1 u1\n") + + def test_model_mount(self): + p = {"spec": {"volumes": [{"name": "home", "emptyDir": {}}, + {"name": "model-cache", "persistentVolumeClaim": {"claimName": "model-store"}}], + "containers": [{"name": "main", "volumeMounts": [ + {"name": "home", "mountPath": "/home"}, + {"name": "model-cache", "mountPath": "/mnt/models", "subPath": "gemma-4-26b"}]}]}} + self.assertEqual(h62.model_mount(p, "main"), "/mnt/models") + with self.assertRaises(ValueError): + h62.model_mount(p, "other") + p["spec"]["volumes"][1]["persistentVolumeClaim"]["claimName"] = "x" + with self.assertRaises(ValueError): + h62.model_mount(p, "main") + + +class VerifyScriptTest(unittest.TestCase): + """B5's weights, read in its pod: every LFS file of the QAT checkpoint at f1e06dc5, and the + commit `hf download --local-dir` recorded for each file.""" + + def setUp(self): + import hashlib + self.tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmp) + self.d = os.path.join(self.tmp, "models") + os.makedirs(os.path.join(self.d, ".cache", "huggingface", "download")) + files = {"model-00001.safetensors": b"w1" * 1000, "model-00002.safetensors": b"w2" * 1000, "config.json": b"{}"} + sib = [] + for n, data in files.items(): + with open(os.path.join(self.d, n), "wb") as f: + f.write(data) + meta = os.path.join(self.d, ".cache", "huggingface", "download", n + ".metadata") + with open(meta, "w") as f: + f.write(h62.PINS["qat_rev"] + "\netag\n1730000000.0\n") + s = {"rfilename": n, "size": len(data)} + if n.endswith(".safetensors"): + s["lfs"] = {"sha256": hashlib.sha256(data).hexdigest(), "size": len(data)} + sib.append(s) + sib.append({"rfilename": "README.md", "size": 10}) # not downloaded: noted + self.sib = {"sha": h62.PINS["qat_rev"], "siblings": sib} + self.script = os.path.join(self.tmp, "verify.py") + with open(self.script, "w") as f: + f.write(h62.verify_script(self.sib)) + + def check(self): + r = subprocess.run([sys.executable, self.script, self.d], capture_output=True, text=True) + return r.returncode, json.loads(r.stdout.splitlines()[-1]) + + def test_matching_weights_pass_and_name_their_commit(self): + rc, s = self.check() + self.assertEqual((rc, s["bad"], s["missing"], s["checked"]), (0, 0, 0, 2)) + self.assertEqual(s["revisions"], [h62.PINS["qat_rev"]]) + self.assertEqual(s["want"], h62.PINS["qat_rev"]) + + def test_a_changed_or_missing_weight_fails(self): + with open(os.path.join(self.d, "model-00002.safetensors"), "ab") as f: + f.write(b"x") + rc, s = self.check() + self.assertEqual((rc, s["bad"]), (1, 1)) + os.unlink(os.path.join(self.d, "model-00001.safetensors")) + rc, s = self.check() + self.assertEqual((rc, s["missing"]), (1, 1)) + + def test_cli_writes_the_script(self): + p = os.path.join(self.tmp, "sib.json") + with open(p, "w") as f: + json.dump(self.sib, f) + r = run("verify-script", p) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(r.stdout, h62.verify_script(self.sib)) + + +class TriggerTest(unittest.TestCase): + """The re-run trigger: an EAS release at v1.2.0 or later (no release candidate) tagged, or an + EAT release at 2.0.0 or later tagged.""" + + EAS = ("1d5686e75f8ec0b1edac206e9ca4db7ddde32250\trefs/tags/v1.2.0-rc.1\n" + "5c27fb39a95eecec41aa4e29a1188984e9c726f3\trefs/tags/v1.2.0-rc.1^{}\n" + "aaaa\trefs/tags/v1.1.0\n") + EAT = "bbbb\trefs/tags/release-1.1.0\ncccc\trefs/tags/release-1.0.3-hotfix\n" + + def test_nothing_fired_today(self): + self.assertEqual(h62.trigger(self.EAS, self.EAT), {"eas": [], "eat": []}) + + def test_ga_or_an_eat_2_release_fires(self): + eas = self.EAS + "dddd\trefs/tags/v1.2.0\ndddd\trefs/tags/v1.2.0^{}\n" + self.assertEqual(h62.trigger(eas, self.EAT)["eas"], ["v1.2.0"]) + self.assertEqual(h62.trigger(self.EAS + "e\trefs/tags/v1.2.1\n", self.EAT)["eas"], ["v1.2.1"]) + self.assertEqual(h62.trigger(self.EAS + "e\trefs/tags/v1.2.0-rc.2\n", self.EAT)["eas"], []) + for tag in ("release-2.0.0", "v2.0.0", "release-2.1.0"): + self.assertEqual(h62.trigger(self.EAS, self.EAT + f"f\trefs/tags/{tag}\n")["eat"], [tag], tag) + + def test_a_later_ga_fires_too(self): + # Full support may ship under a later number than v1.2.x or 2.x; it still fires. + for tag in ("v1.3.0", "v2.0.0", "v1.10.0"): + self.assertEqual(h62.trigger(self.EAS + f"e\trefs/tags/{tag}\n", self.EAT)["eas"], [tag], tag) + for tag in ("v1.3.0-rc.1", "v1.1.1", "v1.0.0", "v2.0.0-beta"): + self.assertEqual(h62.trigger(self.EAS + f"e\trefs/tags/{tag}\n", self.EAT)["eas"], [], tag) + # EAT tags its hotfix releases release-X.Y.Z-hotfix (release-1.0.3-hotfix). + for tag in ("release-3.0.0", "release-2.0.1-hotfix", "v3.1.0"): + self.assertEqual(h62.trigger(self.EAS, self.EAT + f"f\trefs/tags/{tag}\n")["eat"], [tag], tag) + for tag in ("release-1.2.0", "release-2.0.0-rc1", "release-0.5.0"): + self.assertEqual(h62.trigger(self.EAS, self.EAT + f"f\trefs/tags/{tag}\n")["eat"], [], tag) + + def test_cli(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + for n, t in (("eas", self.EAS), ("eat", self.EAT + "f\trefs/tags/release-2.0.0\n")): + with open(os.path.join(tmp, n), "w") as f: + f.write(t) + r = run("trigger", os.path.join(tmp, "eas"), os.path.join(tmp, "eat")) + self.assertEqual(r.returncode, 0) + self.assertEqual(r.stdout, "fired: eat release-2.0.0\n") + + +class CampaignMathTest(unittest.TestCase): + def test_pairs_left(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + self.assertEqual(h62.pairs_left(tmp), 24) + with open(os.path.join(tmp, "pairs.jsonl"), "w") as f: + f.writelines(json.dumps({"j": j, "status": "complete"}) + "\n" for j in range(10)) + f.write('{"j": 10, "sta') # a line still being written + self.assertEqual(h62.pairs_left(tmp), 14) + self.assertEqual(run("pairs-left", tmp).stdout, "14\n") + + H50 = {"within_budget_success": {"A1": {"k": 19, "n": 24, "rate": 19 / 24}, "B1": {"k": 21, "n": 24, "rate": 0.875}}, + "attempt_median_s": {"A1": 8.0, "B1": 12.0}} + H62 = {"within_budget_success": {"A1": {"k": 20, "n": 24, "rate": 20 / 24}, "B5": {"k": 18, "n": 24, "rate": 0.75}}, + "attempt_median_s": {"A1": 9.2, "B5": 15.0}} + + def test_vmtax_is_a_description(self): + t = h62.vmtax(self.H50, self.H62) + self.assertEqual(t["a1_host"], {"success": "19/24", "attempt_median_s": 8.0}) + self.assertEqual(t["a1_guest"], {"success": "20/24", "attempt_median_s": 9.2}) + self.assertAlmostEqual(t["attempt_median_guest_over_host"], 1.15) + self.assertEqual(t["gated"], False) + + def test_vmtax_describes_the_vllm_side_too(self): + # B5 in its guest against H50's B1 on the host: the same vLLM CPU v0.28.0 image and QAT + # weights under another deployment, so the guest and the deployment are read together. + t = h62.vmtax(self.H50, self.H62) + self.assertEqual(t["b1_host"], {"success": "21/24", "attempt_median_s": 12.0}) + self.assertEqual(t["b5_guest"], {"success": "18/24", "attempt_median_s": 15.0}) + self.assertAlmostEqual(t["b_attempt_median_guest_over_host"], 1.25) + self.assertIn("deployment", t["b_note"]) + self.assertEqual(t["gated"], False) + + def test_vmtax_without_a_b5_reading(self): + h62r = {"within_budget_success": {"A1": {"k": 20, "n": 24}}, "attempt_median_s": {"A1": 9.2}} + t = h62.vmtax(self.H50, h62r) + self.assertEqual(t["b5_guest"], None) + self.assertEqual(t["b_attempt_median_guest_over_host"], None) + + +def stub(path, body): + with open(path, "w") as f: + f.write("#!/usr/bin/env bash\n" + body) + os.chmod(path, os.stat(path).st_mode | stat.S_IXUSR) + + +class ArmScriptTest(unittest.TestCase): + """bench/h62/arm.sh against a stub ssh that answers from canned files: the restart wait, + the cgroup spec the idle-arm check reads, cpustat and the ports.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmp) + self.log = os.path.join(self.tmp, "ssh.log") + # The stub logs each remote command, and answers `get pods` from the next pods-N.json. + stub(os.path.join(self.tmp, "ssh"), textwrap.dedent(f'''\ + cmd="${{@: -1}}" + echo "$*" >> "{self.log}" + case "${{cmd}}" in + *"get pods -l"*) + n=$(cat "{self.tmp}/n" 2>/dev/null || echo 0) + f="{self.tmp}/pods-${{n}}.json"; [ -f "$f" ] || f=$(ls "{self.tmp}"/pods-*.json | sort -V | tail -1) + cat "$f"; echo $(( n + 1 )) > "{self.tmp}/n" ;; + *"find /sys/fs/cgroup"*) echo /sys/fs/cgroup/kubepods.slice/kubepods-podu1.slice ;; + *"cat /x/cpu.stat"*) printf 'usage_usec 77\\n' ;; + *) : ;; + esac + ''')) + self.env = dict(os.environ, H62_SSH=os.path.join(self.tmp, "ssh"), H62_STATE=self.tmp, H62_POLL_S="0.1") + + def arm(self, *args): + return subprocess.run(["bash", os.path.join(HERE, "arm.sh"), *args], capture_output=True, text=True, env=self.env) + + def pods(self, *docs): + for i, d in enumerate(docs): + with open(os.path.join(self.tmp, f"pods-{i}.json"), "w") as f: + json.dump({"items": d}, f) + + def test_one_lookup_per_call(self): + self.assertEqual(run("arm-fields", "A1").stdout, + "a h62-a 18831 k3d-gpu-agent app=ovms svc/ovms 8000 ovms \n") + self.assertEqual(h62.arm_fields("B5")[-1], "llminferenceservice/gemma-4-26b") + self.assertEqual(read(HERE, "arm.sh").count("P arm"), 1) # cpustat runs on every idle read + + def test_ports_and_unknown_arm(self): + self.assertEqual(self.arm("A1", "port").stdout, "18831\n") + self.assertEqual(self.arm("B5", "port").stdout, "18832\n") + self.assertEqual(self.arm("C9", "port").returncode, 2) + + def test_restart_waits_for_a_new_ready_pod(self): + self.pods([pod("p0", "u0")], [pod("p0", "u0", deleting=True)], [pod("p1", "u1", ready=False)], [pod("p1", "u1")]) + r = self.arm("B5", "restart", "30") + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn("ready: p1 u1", r.stdout) + log = read(self.log) + self.assertIn("h62-b", log) + self.assertIn("kubectl -n llm-inference delete pod -l app.kubernetes.io/part-of=llminferenceservice," + "app.kubernetes.io/name=gemma-4-26b,kserve.io/component=workload --wait=false", log) + + def test_restart_not_ready_in_time_exits_3(self): + self.pods([pod("p0", "u0")], [pod("p1", "u1", ready=False)]) + self.assertEqual(self.arm("A1", "restart", "1").returncode, 3) + self.assertEqual(self.arm("A1", "restart", "0").returncode, 3) # no time left at all + self.assertIn("kubectl -n k3d-gpu-agent delete pod -l app=ovms --wait=false", read(self.log)) + + def test_cgroup_spec_and_cpustat(self): + self.pods([pod("p1", "u1")]) + r = self.arm("B5", "cgroup") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(r.stdout, f"exec:{os.path.join(HERE, 'arm.sh')} B5 cpustat " + "/sys/fs/cgroup/kubepods.slice/kubepods-podu1.slice\n") + self.assertIn("-name podu1 -o -name '*podu1.slice'", read(self.log)) + r = self.arm("B5", "cpustat", "/x") + self.assertEqual(r.stdout, "usage_usec 77\n") + + def test_forward_restarts_the_guest_port_forward(self): + r = self.arm("A1", "forward") + self.assertEqual(r.returncode, 0, r.stderr) + log = read(self.log) + self.assertIn("port-forward --address 0.0.0.0 svc/ovms 18831:8000", log) + self.assertIn("pf-A1.pid", log) + self.arm("A1", "unforward") + self.assertIn("kill", read(self.log).splitlines()[-1]) + + +class ScriptsTest(unittest.TestCase): + """The shell scripts: syntax, and what each one is pinned to run.""" + + SCRIPTS = ("arm.sh", "vm.sh", "guest-a.sh", "guest-b.sh", "setup.sh", "smoke.sh", "pairs-hold.sh", + "campaign.sh", "teardown.sh", "upstream.sh") + + def test_syntax(self): + for s in self.SCRIPTS: + r = subprocess.run(["bash", "-n", os.path.join(HERE, s)], capture_output=True, text=True) + self.assertEqual(r.returncode, 0, f"{s}: {r.stderr}") + + def test_shellcheck_when_present(self): + if not shutil.which("shellcheck"): + self.skipTest("no shellcheck") + r = subprocess.run(["shellcheck", "-S", "warning", *(os.path.join(HERE, s) for s in self.SCRIPTS)], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stdout) + + def test_vm_up_runs_the_argv_h62_py_builds(self): + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + with open(os.path.join(tmp, "id_ed25519"), "w") as f: # a key exists: none is made + f.write("private\n") + with open(os.path.join(tmp, "id_ed25519.pub"), "w") as f: + f.write(CloudInitTest.KEY + "\n") + env = dict(os.environ, H62_DRY="1", H62_STATE=tmp, H62_KVM_GID="992") + r = subprocess.run(["bash", os.path.join(HERE, "vm.sh"), "up", "b"], capture_output=True, text=True, env=env) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn("docker run -d --name h62-vm-b --network host --device /dev/kvm", r.stdout) + self.assertIn("-smp 96,sockets=1,cores=48,threads=2 -m 192G", r.stdout) + want = h62.qemu_argv("b", tmp, h62.QEMU_IMAGE, uid=os.getuid(), gid=os.getgid(), kvm_gid=992) + self.assertIn(" ".join(want) + "\n", r.stdout) + self.assertIn("qemu-img create -f qcow2 -F qcow2 -b /h62/base.img /h62/b/disk.qcow2 400G", r.stdout) + self.assertIn("genisoimage -output /h62/b/seed.iso -volid cidata -joliet -rock", r.stdout) + self.assertEqual(read(tmp, "b", "meta-data"), h62.meta_data("b")) + self.assertEqual(read(tmp, "b", "user-data"), h62.user_data("b", CloudInitTest.KEY)) + self.assertEqual(read(tmp, "ssh_config"), h62.ssh_config(tmp)) + + def test_pairs_hold_is_h50s_hold_with_remote_arms(self): + sh = read(HERE, "pairs-hold.sh") + self.assertIn('crescent/hold.sh" "${name}" 2400 env', sh) + self.assertIn("hold) name=H62-hold${N} R=${B}/results/H62 guard=600 ;;", sh) + for item in ("LIVE_MODE=pairs", "LIVE_CAP=2370", 'LIVE_HOLD="${N}"', "LIVE_MODEL_SET=gemma4", + 'bash "${HERE}/../eatflow/live.sh" H62', "SERVING_VMS=h62-vm-a,h62-vm-b"): + self.assertIn(item, sh, item) + for absent in ("LIVE_MOUNT_B", "LIVE_EAT_CHART", "K3D_GPU_IMAGE"): # nothing is deployed on the host + self.assertNotIn(absent, sh, absent) + + def test_live_sh_calls_the_remote_arm_as_one_word(self): + # LIVE_REMOTE_A names a script and its arm: live.sh calls "${RA}" with the arm as one word, + # so arm.sh is called through a two-line wrapper per arm. + for arm in ("A1", "B5"): + w = read(HERE, f"arm-{arm}") + self.assertIn(f'exec "$(dirname "$0")/arm.sh" {arm} "$@"', w) + self.assertTrue(os.access(os.path.join(HERE, f"arm-{arm}"), os.X_OK)) + sh = read(HERE, "pairs-hold.sh") + self.assertIn('LIVE_REMOTE_A="${HERE}/arm-A1"', sh) + self.assertIn('LIVE_REMOTE_B="${HERE}/arm-B5"', sh) + + def test_guest_b_is_intels_documented_path(self): + sh = read(HERE, "guest-b.sh") + steps = ['git clone -q -b "${TAG}" https://github.com/intel/enterprise-ai-solutions.git', + 'h62.py" eas-pin configs/repos/repos.agent-toolkit.yaml', + "./es_auto_installer.sh configure --force", + "./es_auto_installer.sh init agent-toolkit", + 'h62.py" eas-edit env/local', + "./es_auto_installer.sh install agent-toolkit"] + pos = [sh.index(s) for s in steps] + self.assertEqual(pos, sorted(pos), "steps in the documented order") + self.assertIn('[ "$(git rev-parse HEAD)" = "${SHA}" ]', sh) + # prerequisites.md "Node Kernel Settings", verbatim values. + self.assertIn("fs.inotify.max_user_instances = 8192", sh) + self.assertIn("fs.inotify.max_user_watches = 1048576", sh) + # global_config.yaml (base_domain_name and the rest) is only copied into the records. + uses = [ln.strip() for ln in sh.splitlines() if "global_config.yaml" in ln] + self.assertEqual(uses, ["# base_domain_name stays EAS's default: global_config.yaml is not edited", + "for f in global_config.yaml config.agent-toolkit.yaml config.inference.yaml " + "models.yaml inventory/hosts.yaml; do"]) + self.assertIn("HF_TOKEN", sh) # recorded as set or unset, never printed + self.assertNotIn("--only", sh) + self.assertNotIn("existing_kubernetes", sh) + + def test_guest_a_deploys_h50s_a1(self): + sh = read(HERE, "guest-a.sh") + live = read(REPO, "bench", "eatflow", "live.sh") + for item in ("K3D_GPU_AGENT_DEVICE=CPU", "K3D_GPU_AGENT_MODEL=k3dgpu/gemma-4-26B-A4B-it-qat-int4-ov", + "K3D_GPU_AGENT_MODEL_NAME=gemma-4-26b-a4b-it", "K3D_GPU_AGENT_DRAFT=off", + "K3D_GPU_AGENT_TOOL_PARSER=gemma4", "K3D_GPU_AGENT_REASONING_PARSER=''", "K3D_GPU_AUTOQUANT=off"): + self.assertIn(item, sh, item) + # The same model set live.sh gave A1 in H50. + self.assertIn("TGT=k3dgpu/gemma-4-26B-A4B-it-qat-int4-ov TGT_DIR=gemma-4-26B-A4B-it-qat-int4-ov A_TOOL=gemma4 A_DRAFT=off", live) + self.assertIn("ALIAS=google/gemma-4-26B-A4B-it SERVED=gemma-4-26b-a4b-it", live) + self.assertIn('stage.py" ovms', sh) + self.assertIn("K3D_GPU_CLUSTER=k3dgpu-e-h62", sh) + self.assertIn('K3D_GPU_IMAGE="$(P pin node_image)"', sh) + self.assertIn("eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1%", sh) + for pkg in ("docker-ce", "docker-ce-cli"): + self.assertIn(f'{pkg}="$(P pin docker_ce)"', sh) + self.assertIn('containerd.io="$(P pin containerd)"', sh) + self.assertIn("docker_gpg_fpr", sh) + + def test_smoke_gates_b5_through_eas_itself(self): + sh = read(HERE, "smoke.sh") + self.assertIn("./es_auto_installer.sh validate agent-toolkit", sh) + self.assertIn("EAT_MODE=eas", sh) + self.assertIn("MODEL=$(P pin catalog_name)", sh) + self.assertIn("EAT_MODE=eas EAT_MODEL=${MODEL}", sh) + self.assertIn('"${HERE}/pairs-hold.sh" 0 smoke', sh) + hold = read(HERE, "pairs-hold.sh") + self.assertIn("smoke) name=H62-smoke", hold) + self.assertIn("guard=99999", hold) + + def test_campaign_order_and_caps(self): + sh = read(HERE, "campaign.sh") + pos = [sh.index(s) for s in ('"${HERE}/setup.sh"', '"${HERE}/smoke.sh"', '"${HERE}/pairs-hold.sh" "${n}"', + "pairs-report")] + self.assertEqual(pos, sorted(pos)) + self.assertIn("MAX_HOLDS=8", sh) + self.assertIn('h62.py" pairs-left', sh) + self.assertIn("trap teardown EXIT", sh) + self.assertIn("--hyp H62 --arm-a A1 --arm-b B5", sh) + self.assertIn("NOT_DELIVERABLE_MAX=2", sh) + + def test_setup_caps_and_gates(self): + sh = read(HERE, "setup.sh") + self.assertIn('flock -w 7200 9', sh) + self.assertIn("A_CAP=2700", sh) # 45 min + self.assertIn("B_CAP=7200", sh) # 120 min + for phase in ("host", "guest", "nfd", "weights"): + self.assertIn(f"P gates {phase}", sh) + + +if __name__ == "__main__": + unittest.main(verbosity=1) diff --git a/bench/h62/upstream.sh b/bench/h62/upstream.sh new file mode 100755 index 00000000..38fe0d4e --- /dev/null +++ b/bench/h62/upstream.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# upstream.sh: H62's re-run trigger (EXPERIMENTS.md "H62 pre-registration", "Re-run trigger"). +# Reads the tags of EAS and EAT with `git ls-remote` and prints "fired: none", or the release tags +# that fired it: an EAS v1.2.x release (not a release candidate) or an EAT 2.x release. +set -euo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +d=$(mktemp -d) +trap 'rm -rf "${d}"' EXIT +git ls-remote --tags https://github.com/intel/enterprise-ai-solutions.git > "${d}/eas" +git ls-remote --tags https://github.com/intel/enterprise-agent-toolkit.git > "${d}/eat" +python3 "${HERE}/h62.py" trigger "${d}/eas" "${d}/eat" diff --git a/bench/h62/vm.sh b/bench/h62/vm.sh new file mode 100755 index 00000000..f6cb067d --- /dev/null +++ b/bench/h62/vm.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# vm.sh build | fetch | up VM | wait VM SECONDS | facts VM | ssh VM CMD... | down VM | purge +# H62's two KVM guests on crescent (EXPERIMENTS.md "H62 pre-registration"): VM a serves A1, VM b +# serves B5. QEMU runs in the pinned container of qemu.Dockerfile, so the host gets no package. +# Disks, seed images, the ssh key and ssh_config live under H62_STATE (default +# ~/k3dgpu-bench/h62); results do not. Shapes, ports and pins come from h62.py. +# build the QEMU image (h62.py qemu-image); its package list into H62_STATE +# fetch the Ubuntu 24.04 cloud image into H62_STATE/base.img, checked against its sha256 +# up VM the guest's qcow2 overlay, its cloud-init seed and its QEMU container; VM a also +# gets A1's IR read-only (H62_IR, default the IR live.sh's gemma4 set names) +# wait VM SECONDS until the guest answers ssh and cloud-init has finished +# facts VM one JSON line: nproc, amx_tile count, MemTotal and kernel (the guest gate's facts) +# ssh VM CMD... a command in the guest +# down VM power the guest off (at most 120 s), then remove its container +# purge the guests' disks, the cloud image and the QEMU image (refused while a guest runs) +# H62_DRY=1 prints each docker command instead of running it. +set -euo pipefail +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +P() { python3 "${HERE}/h62.py" "$@"; } +STATE=${H62_STATE:-${HOME}/k3dgpu-bench/h62} +IMAGE=$(P qemu-image) +DRY=${H62_DRY:-0} +run() { if [ "${DRY}" = 1 ]; then echo "$*"; else "$@"; fi; } +in_qemu() { run docker run --rm --user "$(id -u):$(id -g)" -v "${STATE}:/h62" "${IMAGE}" "$@"; } +gssh() { ssh -F "${STATE}/ssh_config" "$(P vm "$1" host)" "${@:2}"; } +exists() { [ "${DRY}" != 1 ] && docker ps -a --format '{{.Names}}' | grep -x "$1" > /dev/null; } +mkdir -p "${STATE}" +chmod 700 "${STATE}" +case ${1:?usage: vm.sh build|fetch|up VM|wait VM SECONDS|facts VM|ssh VM CMD...|down VM|purge} in +build) + run docker build -t "${IMAGE}" -f "${HERE}/qemu.Dockerfile" "${HERE}" + if [ "${DRY}" != 1 ]; then + docker image inspect --format '{{.Id}}' "${IMAGE}" > "${STATE}/qemu-image.txt" + docker run --rm "${IMAGE}" cat /h62-packages.txt > "${STATE}/qemu-packages.txt" + fi ;; +fetch) + if [ ! -f "${STATE}/base.img" ]; then + curl -fsSLo "${STATE}/base.img.part" "$(P pin cloud_image_url)" + mv "${STATE}/base.img.part" "${STATE}/base.img" + fi + echo "$(P pin cloud_image_sha256) ${STATE}/base.img" | sha256sum -c + [ "$(stat -c %s "${STATE}/base.img")" = "$(P pin cloud_image_bytes)" ] || { echo "base.img: wrong size"; exit 1; } + chmod 444 "${STATE}/base.img" ;; +up) + vm=${2:?usage: vm.sh up a|b} + name=$(P vm "${vm}" name) + if exists "${name}"; then echo "${name} exists: vm.sh down ${vm} first"; exit 1; fi + d=${STATE}/${vm} + mkdir -p "${d}" + [ -f "${STATE}/id_ed25519" ] || ssh-keygen -q -t ed25519 -N '' -C h62 -f "${STATE}/id_ed25519" + P ssh-config "${STATE}" > "${STATE}/ssh_config" + # A new guest has a new host key on the same forwarded port. + [ ! -f "${STATE}/known_hosts" ] || ssh-keygen -q -R "[127.0.0.1]:$(P vm "${vm}" ssh_port)" \ + -f "${STATE}/known_hosts" > /dev/null 2>&1 || true + P user-data "${vm}" "${STATE}/id_ed25519.pub" > "${d}/user-data" + P meta-data "${vm}" > "${d}/meta-data" + rm -f "${d}/seed.iso" "${d}/disk.qcow2" + in_qemu genisoimage -output "/h62/${vm}/seed.iso" -volid cidata -joliet -rock "/h62/${vm}/user-data" "/h62/${vm}/meta-data" + in_qemu qemu-img create -f qcow2 -F qcow2 -b /h62/base.img "/h62/${vm}/disk.qcow2" "$(P vm "${vm}" disk_gib)G" + share=() + if [ "$(P vm "${vm}" share)" = True ]; then + share=("$(readlink -f "${H62_IR:-${HOME}/k3dgpu-bench/models/gemma-4-26B-A4B-it-qat-int4-ov}")") + fi + kvm=${H62_KVM_GID:-$(stat -c %g /dev/kvm)} + mapfile -t argv < <(P qemu-argv "${vm}" "${STATE}" "${IMAGE}" "$(id -u)" "$(id -g)" "${kvm}" "${share[@]}") + [ "${#argv[@]}" -gt 10 ] || { echo "no QEMU command line for VM ${vm}"; exit 1; } + run "${argv[@]}" ;; +wait) + vm=${2:?usage: vm.sh wait a|b SECONDS} + end=$(( SECONDS + ${3:?usage: vm.sh wait a|b SECONDS} )) + until gssh "${vm}" true > /dev/null 2>&1; do + [ "${SECONDS}" -lt "${end}" ] || { echo "VM ${vm}: no ssh within ${3} s"; exit 1; } + sleep 5 + done + gssh "${vm}" cloud-init status --wait > /dev/null ;; +facts) + gssh "${2:?usage: vm.sh facts a|b}" python3 - <<'PY' +import json, os, platform +flags = [ln for ln in open("/proc/cpuinfo") if ln.startswith("flags")] +mem = {k: v for k, _, v in (ln.partition(":") for ln in open("/proc/meminfo"))} +print(json.dumps({"nproc": os.cpu_count(), "amx_tile": sum(" amx_tile" in ln for ln in flags), + "mem_total_kib": int(mem["MemTotal"].split()[0]), "kernel": platform.release()})) +PY + ;; +ssh) + gssh "${2:?usage: vm.sh ssh a|b CMD...}" "${@:3}" ;; +down) + vm=${2:?usage: vm.sh down a|b} + name=$(P vm "${vm}" name) + gssh "${vm}" sudo poweroff > /dev/null 2>&1 || true + for _ in $(seq 60); do + [ "$(docker inspect -f '{{.State.Running}}' "${name}" 2>/dev/null)" = true ] || break + sleep 2 + done + run docker rm -f "${name}" > /dev/null 2>&1 || true ;; +purge) + for vm in a b; do + if exists "$(P vm "${vm}" name)"; then echo "VM ${vm} still exists: vm.sh down ${vm} first"; exit 1; fi + done + rm -rf "${STATE:?}/a" "${STATE:?}/b" "${STATE}/base.img" "${STATE}/base.img.part" "${STATE}/known_hosts" + run docker image rm "${IMAGE}" > /dev/null 2>&1 || true ;; +*) + echo "usage: vm.sh build|fetch|up VM|wait VM SECONDS|facts VM|ssh VM CMD...|down VM|purge"; exit 2 ;; +esac diff --git a/docs/intel/technical-report.md b/docs/intel/technical-report.md index 9eb754af..c2fc55c3 100644 --- a/docs/intel/technical-report.md +++ b/docs/intel/technical-report.md @@ -1,17 +1,17 @@ # k3d-gpu on Intel hardware: technical report -Status: working report, 2026-10-29. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp and H54q runs (81 rows). +Status: working report, 2026-10-29. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp, H54q and H53 runs (82 rows). ## Abstract k3d-gpu turns one Linux host into a single-node k3s cluster (k3d) that detects the host's Intel hardware and configures it. It sets NRI balloons for hybrid P/E and NUMA CPUs, device plugins for iGPU, dGPU, NPU, DSA, IAA, QAT and Gaudi, and placement labels for agent and inference pods. -This report covers what was measured, and how. Every result but H52, H52b, H25x, H27x, H28x, H34x, H50, H50o, H51, H54a, H54d, H54q, H54b, H54c and H56 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. +This report covers what was measured, and how. Every result but H52, H52b, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54q, H54b, H54c and H56 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. - Seventeen hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x is H25 replicated on the Xeon 6741P. H57dm (the Gemma 4 QAT MTP drafter speeds llama.cpp decode on the P-cores) is the newest. - Seventeen are REFUTED. Six of them changed shipped defaults. The newest, H52b, is H52 after `k3d-gpu host-setup --apply` on the Xeon 6741P: DSA was advertised and reached a pod, but IAA got no work queue, the QAT plugin advertised none of the VFs, and `up` advertised two of the four B70s: the serving pairs hold about half the VRAM of every card, and its busy-card filter (half or more) left out the two just over half and kept the two just under it [ledger:H52b]. Before it, H51 ran the graded agents-per-box ramp on the Xeon 6741P: the k3d-gpu CPU stack failed at one agent (turn p95 10.41 s against the 10 s SLO), and so did EAT's stock vLLM CPU arm [ledger:H51]. H50 ran inside EAT's banking flow on the same host: EAT's current stock vLLM CPU arm (release-1.1.0) delivered at least as many tasks within the turn budget as the k3d-gpu CPU stack: 21 of 24 against 19 of 24 [ledger:H50]. -- 31 are INCONCLUSIVE, mostly because the shared host throttles or its load was unmatched. H54q (the QAT-matched int4 IR of Gemma 4 26B-A4B against the bf16 QAT weights and OpenVINO's base-model IR, on GSM8K items 0-199 on the Xeon 6741P) is INCONCLUSIVE: the QAT-matched IR stayed inside the pre-registered margin of the bf16 reference, but the base-model IR scored no lower than it, with every build near the item set's ceiling. H50o (the k3d-gpu CPU stack against EAT's own OVMS chart on the Xeon 6741P) is INCONCLUSIVE because that chart's OVMS 2025.4 crashed on loading Gemma 4 26B-A4B, so no pair ran. H54a and H54d (Gemma 4 26B-A4B QAT on the Xeon 6741P) are INCONCLUSIVE because the stock arm, EAT 1.0.3's vLLM CPU v0.24.0, as this harness rendered EAT's entry (no CPU balloon, requests or limits), stalled in both of H54a's holds, so no arm could be scored against it. EAT 1.1.0's vLLM CPU v0.28.0 was not measured in H54a; H50 ran it in EAT's banking flow. H54b and H54c (one Arc Pro B70 against two, and experts in DRAM with the rest on one B70) are INCONCLUSIVE because their "one card" pods saw all four cards, so no arm was delivered as pre-registered. H57da (k3d-gpu's OVMS pick vs llama.cpp for Gemma 4 QAT on the i9-13900T) did not resolve: the strictly pinned llama.cpp arms tracked host load; H57db did not run (memory). H57dp and H57de are INCONCLUSIVE: they re-ran that comparison with an unpinned llama.cpp arm, and the load gate failed on OVMS's low foreign P-core load (0.45 CPU-s/s against the llama.cpp arms' 3.1-3.4); the unpinned arm was 1.108x the pinned one and 1.369x OVMS, not gated, and OVMS ran without its P-core balloon. H28x (H28 replicated on the Xeon 6741P) is INCONCLUSIVE: its one pre-registered hold failed in setup and measured nothing; one re-run is registered. H56 (two holds at once in cpuset slices of the Xeon 6741P) is INCONCLUSIVE: in every hold with a sibling, the sibling's banking block ended shortly after the measured block began, so all four were VOID and no interference statistic was scored; crescent rows stay at one hold at a time. -- 16 are PENDING: the Xeon 6741P rows H53, H60 (Qwen3.8-Flash-Next from host DRAM) and the zero-touch thesis row H58, the cross-host replication H7x, the engine-choice rows H59 (Xeon 6741P), H59d (i9-13900T) and H59w (i7-13700), H55 (SLO-gated admission of an E-core neighbour on the i9-13900T), and the second client host's (wildct, i7-13700) rows H52w, H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. +- 32 are INCONCLUSIVE, mostly because the shared host throttles or its load was unmatched. H54q (the QAT-matched int4 IR of Gemma 4 26B-A4B against the bf16 QAT weights and OpenVINO's base-model IR, on GSM8K items 0-199 on the Xeon 6741P) is INCONCLUSIVE: the QAT-matched IR stayed inside the pre-registered margin of the bf16 reference, but the base-model IR scored no lower than it, with every build near the item set's ceiling. H50o (the k3d-gpu CPU stack against EAT's own OVMS chart on the Xeon 6741P) is INCONCLUSIVE because that chart's OVMS 2025.4 crashed on loading Gemma 4 26B-A4B, so no pair ran. H54a and H54d (Gemma 4 26B-A4B QAT on the Xeon 6741P) are INCONCLUSIVE because the stock arm, EAT 1.0.3's vLLM CPU v0.24.0, as this harness rendered EAT's entry (no CPU balloon, requests or limits), stalled in both of H54a's holds, so no arm could be scored against it. EAT 1.1.0's vLLM CPU v0.28.0 was not measured in H54a; H50 ran it in EAT's banking flow. H54b and H54c (one Arc Pro B70 against two, and experts in DRAM with the rest on one B70) are INCONCLUSIVE because their "one card" pods saw all four cards, so no arm was delivered as pre-registered. H53 (EAT's banking flow on the B70s: k3d-gpu's one-card OVMS pick against EAT 1.1.0's vLLM XPU path across all four cards) is INCONCLUSIVE because the EAT arm did not start in a k3d-gpu cluster: oneCCL needs `/dev/dri/by-path` for its cross-card collective, and the k3d node has none, so no hold ran. H57da (k3d-gpu's OVMS pick vs llama.cpp for Gemma 4 QAT on the i9-13900T) did not resolve: the strictly pinned llama.cpp arms tracked host load; H57db did not run (memory). H57dp and H57de are INCONCLUSIVE: they re-ran that comparison with an unpinned llama.cpp arm, and the load gate failed on OVMS's low foreign P-core load (0.45 CPU-s/s against the llama.cpp arms' 3.1-3.4); the unpinned arm was 1.108x the pinned one and 1.369x OVMS, not gated, and OVMS ran without its P-core balloon. H28x (H28 replicated on the Xeon 6741P) is INCONCLUSIVE: its one pre-registered hold failed in setup and measured nothing; one re-run is registered. H56 (two holds at once in cpuset slices of the Xeon 6741P) is INCONCLUSIVE: in every hold with a sibling, the sibling's banking block ended shortly after the measured block began, so all four were VOID and no interference statistic was scored; crescent rows stay at one hold at a time. +- 16 are PENDING: the Xeon 6741P rows H60 (Qwen3.8-Flash-Next from host DRAM), the zero-touch thesis row H58 and H62 (the k3d-gpu CPU stack against Intel's new default, EAS v1.2.0-rc.1 installing EAT 2.0, each in its own KVM guest), the cross-host replication H7x, the engine-choice rows H59 (Xeon 6741P), H59d (i9-13900T) and H59w (i7-13700), H55 (SLO-gated admission of an E-core neighbour on the i9-13900T), and the second client host's (wildct, i7-13700) rows H52w, H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. - Every other Intel family is detected and configured in fixture tests only. None has a measurement. No world-first, superlative or priority claim is made; [docs/provenance.md](../provenance.md) supports none. @@ -109,6 +109,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p | H54q | Gemma 4 26B-A4B on the Xeon 6741P, GSM8K items 0-199: the QAT-matched int4 IR (INT4_SYM group 32, from Google's QAT weights) against the bf16 QAT reference and OpenVINO's published base-model IR. The QAT-matched IR stayed inside the pre-registered non-inferiority margin of the bf16 reference [ledger:H54q] | The base-model IR scored no lower than the QAT-matched IR; every build sat near the item set's ceiling, too few items differed to separate them | | H56 | Two A1 banking holds at once in disjoint cpuset slices of the Xeon 6741P. All 8 holds ran; the 4 holds without a sibling are valid. No interference finding [ledger:H56] | Every hold with a sibling is VOID: a B24 block ends at its 24th task, so the sibling's block ended shortly after the measured one began. 4 VOID exceed the cap of 2 | | H54b | Qwen3.8-27B on Arc Pro B70: the best one-card arm against EAT #21's two-card XPU path. Nothing one-card was delivered: the arms on Qwen bf16 or RedHat int4 weights did not fit the disk, two OVMS arms failed on runner bugs, and llama.cpp SYCL ran on all four cards (direction only) [ledger:H54b] | Placement failure (the pinned pod saw every card) and no gate reference | +| H53 | EAT's banking flow on four Arc Pro B70s with Qwen3.8-27B: k3d-gpu's own GPU pick (OVMS int4 on one card) deployed and answered its warm-up. EAT 1.1.0's vLLM XPU path across four cards needed one fix to start (its command sources a oneAPI script the image lacks), then failed in its first cross-card collective: oneCCL opens `/dev/dri/by-path`, which the k3d node and its pods do not have [ledger:H53] | Not deliverable: the EAT arm's pre-registered boot ladder ran out, so no hold ran. The `/dev/dri/by-path` gap is k3d-gpu's | | H54c | Gemma 4 QAT with routed experts in the Xeon's DRAM and the rest on a B70, against a byte-split model. The experts did run on the Xeon, but every GPU arm split its layers over four cards, so the one-card model was never tested (direction only) [ledger:H54c] | Placement failure; the CPU-only input is confounded too | The external bake-offs are in the tables above: the Ollama, llama.cpp and plain OVMS ones are all decided.