From 1f39488be7695dc0d1848347bf1c91f04025bc55 Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Thu, 29 Oct 2026 17:13:20 -0700 Subject: [PATCH 1/2] ledger(H53): amendment R1, the solo runner fix, setup, wiring smokes and the GPU window Amendment R1 is recorded before any H53 hold or smoke. It changes mechanics only: no arm, model, sampling, prediction, statistic, gate, threshold, seed or order changes, and no cap rises. - live.sh: stage_path moves to top level. The solo path called it while it was defined only inside the pairs branch, so AG would have served an empty hostPath. The solo branch also passes LIVE_PAIRS_GUARD as --guard-s, so a wiring smoke runs only the warm-up. - solo.py: alive_blocks, the blocks that can still count after each hold. The window stops once fewer than three can, since the verdict is then fixed at INCONCLUSIVE. - prehold.sh: the window steps. E4's setup.sh under the bench lock, a model re-verify, front-door checks, stop and start through gpu-serving.sh (never rm), the AG and B2 smokes, the scored holds, per-card Prometheus records and a deadman that restores serving if the session dies. --- EXPERIMENTS.md | 115 ++++++- bench/eatflow/live.sh | 29 +- .../results/2026-10-20/H53/prehold/prehold.sh | 304 ++++++++++++++++++ bench/eatflow/solo.py | 26 ++ bench/eatflow/test_pairs.py | 62 ++++ bench/eatflow/test_solo.py | 26 ++ 6 files changed, 549 insertions(+), 13 deletions(-) create mode 100755 bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index c9689b5f..799802c1 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -70,7 +70,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H51 | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) Graded agents per box on the Xeon 6741P: H45's ramp, gates and scoring (B15-RO, typical load, 300 s steps, pooled p95 turn <= 10 s, success >= the arm's N=1 success - 10 points, errors <= 1%, >= 60 window requests), extended to N=16, for A1 and B1. | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement. A1 in [1, 8] (0.5). B1 in [0, 4] (0.5). B1 <= A1 (0.55). | H45's `apb` runner and scorer (`apb-score --hyp H51`, ramp 1, 2, 4, 8, 16), one fresh cluster per hold, at most 4 holds of at most 40 min. A prediction holds when the arm's bracket lies inside its range and is refuted when it lies outside; B1 <= A1 is decided only on matched load. REFUTED when any prediction is refuted, SUPPORTED when all three hold, else INCONCLUSIVE. Runs only if H50's holds end within their cap. Model and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20). B1 as H50, from [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); deploy commit, M1's ranges in the scorer, one hold per arm and the wiring smokes per [H51 amendment S1](#h51-amendment-s1-deploy-commit-scorer-ranges-ramp-per-hold-and-wiring-smokes-2026-10-29). Result ([H51 run](#h51-run-xeon-6741p-2026-10-29)): one hold per arm, 2 of the 4 permitted, none VOID. Both arms failed N=1. A1: 65 window requests, turn p95 10.41 s against the 10 s SLO, success 30/32. B1: 58 window requests (under 60), p95 11.52 s, success 27/29. Agents per box A1 0, outside its predicted [1, 4], so REFUTED. B1's 0 lies inside [0, 4], and B1 <= A1 holds on matched load (0 <= 0). | REFUTED | | H52 | A plain `k3d-gpu up` (no flags, the auto-adapt default) on the Xeon 6741P host as found detects and enables every engine actually present and advertises each usable one to the scheduler, with nothing missing and nothing phantom; engines present but not configured on the host are named, not advertised. | Written 2026-10-20 from an inventory read from sysfs, lspci and CPUID, not from k3d-gpu. 4 x Arc Pro B70 (xe) as `gpu.intel.com/xe: 4`; AMX TILE, BF16, INT8, FP16 and 14 AVX-512/VNNI labels; no hybrid, NUMA, NVIDIA, Gaudi or NPU objects; DSA, IAA and QAT named as present but unconfigured (0.4: `up` may not name them). | The table in [H50-H52 pre-registration](#h50-h52-pre-registration-the-xeon-6741p-2026-10-20): `k3d-gpu doctor`, `K3D_GPU_CLUSTER=k3dgpu-h52 k3d-gpu up`, `kubectl get node -o json`, `k3d-gpu test`, and one enumeration pod per B70. SUPPORTED: every row matches. REFUTED: any row does not. Run 2026-10-20 ([H52 run](#h52-run-xeon-6741p-2026-10-20)): the four B70s (`gpu.intel.com/xe: 4`, `xmx=true`), AMX (TILE, BF16, INT8, FP16) and all 14 AVX-512/VNNI labels detected and reached pods, nothing phantom; but no media engine label on `xe`, and `up` does not name the unconfigured DSA, IAA and QAT (`doctor` does). | REFUTED | | H52b | After `k3d-gpu host-setup --apply` configures the host's DSA, IAA and QAT (the fix that adds that path, merged first), a plain `k3d-gpu up` also advertises them, and each one reaches a pod; the host is then restored exactly. | Written 2026-10-20, before the fix exists. DSA, IAA and QAT allocatable > 0, each at the count host-setup reports configuring; their test pods see `/dev/dsa`, `/dev/iax`, `/dev/vfio` (0.6). The other rows read as H52's expected table, media labels included ([amendment D1](#h52-amendment-d1-the-fixes-and-h52b-2026-10-20)). | Runs after H52, once the host-setup fix is merged; same procedure plus `host-setup --apply` before `up` and its printed revert after. SUPPORTED: the three engines advertised and reached, the H52 rows unchanged, and the host inventory read back equal to the one taken before. REFUTED: any of these fails. The host-setup command (`--apply --with-kernel-wqs`), the plan read first, the counts, the plain `up`'s node image and the revert per [amendment D2](#h52-amendment-d2-h52bs-host-setup-step-on-crescent-2026-10-29); the runner and deploy commit per [amendment D3](#h52-amendment-d3-h52bs-runner-and-deploy-commit-2026-10-29). Result ([H52b run](#h52b-run-xeon-6741p-2026-10-29)): one hold at k3d-gpu 0.1.79; the plan passed D2's gate (8 commands) and applied. DSA advertised 3 of the 3 configured work queues and its pod saw `/dev/dsa`; IAA got no work queue and no resource (as D2 item 4 disclosed); QAT got 48 VFs, but its plugin crash-looped and advertised none; `gpu.intel.com/xe` read 2, not 4: the serving pairs hold about half the VRAM of all four cards, and `up`'s busy-card filter (half the VRAM) left out the two just over half (16.1, 16.2 GiB) and kept the two just under it (15.4, 15.5 GiB), which the pairs also hold, so two enumeration pods ran, not four. Media labels (`vcs` 2, `vecs` 2), AMX, the 14 AVX-512/VNNI labels, CPU 96 and no other vendor read as expected. The third inventory equals the first. Three clauses fail, so REFUTED. | REFUTED | -| H53 | Inside EAT's banking flow on the Arc Pro B70s with Qwen3.8-27B (thinking off), k3d-gpu's own GPU choice AG (OVMS int4 on one card, `agent manifest`, auto device) delivers far more B24 tasks within the 120 s turn budget than B2, "EAT PR #21 nearest profile (Qwen3.6-27B) on Qwen3.8 weights, fp16" (vLLM XPU v0.28.0, TP 4). A stack-vs-stack claim. | Written 2026-10-20, before any H53 measurement. AG >= 0.70 (0.6); B2 boots by ladder rung (b) (0.6); B2 >= 0.50 (0.5); block mean AG - B2 in [-0.15, +0.30] (0.5). INCONCLUSIVE (0.6), REFUTED (0.3), SUPPORTED (0.1). | Solo holds in the order AG, B2, B2, AG, AG, B2 (up to 8), blocks = hold pairs (2k-1, 2k), block-level inference: mean of per-block d over >= 3 counted blocks, t-interval. SUPPORTED: lower bound >= +30 points and AG >= 0.6, L3 gates passed. REFUTED: upper bound < +30 points. Else INCONCLUSIVE. `bench/eatflow/crescent/solo-hold.sh N ARM`, `agentbench solo-report --hyp H53 --arm-a AG --arm-b B2`. See [H53 pre-registration](#h53-pre-registration-eats-banking-flow-on-arc-pro-b70-2026-10-20). | PENDING | +| H53 | Inside EAT's banking flow on the Arc Pro B70s with Qwen3.8-27B (thinking off), k3d-gpu's own GPU choice AG (OVMS int4 on one card, `agent manifest`, auto device) delivers far more B24 tasks within the 120 s turn budget than B2, "EAT PR #21 nearest profile (Qwen3.6-27B) on Qwen3.8 weights, fp16" (vLLM XPU v0.28.0, TP 4). A stack-vs-stack claim. | Written 2026-10-20, before any H53 measurement. AG >= 0.70 (0.6); B2 boots by ladder rung (b) (0.6); B2 >= 0.50 (0.5); block mean AG - B2 in [-0.15, +0.30] (0.5). INCONCLUSIVE (0.6), REFUTED (0.3), SUPPORTED (0.1). | Solo holds in the order AG, B2, B2, AG, AG, B2 (up to 8), blocks = hold pairs (2k-1, 2k), block-level inference: mean of per-block d over >= 3 counted blocks, t-interval. SUPPORTED: lower bound >= +30 points and AG >= 0.6, L3 gates passed. REFUTED: upper bound < +30 points. Else INCONCLUSIVE. `bench/eatflow/crescent/solo-hold.sh N ARM`, `agentbench solo-report --hyp H53 --arm-a AG --arm-b B2`. See [H53 pre-registration](#h53-pre-registration-eats-banking-flow-on-arc-pro-b70-2026-10-20). Node image, deploy commit, order and the restore check per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); the solo runner fix, setup, the wiring smokes, the window and its futility stop per [H53 amendment R1](#h53-amendment-r1-the-solo-runner-fix-setup-wiring-smokes-and-the-gpu-window-2026-10-30); tools per [amendment E4](#crescent-amendment-e4-kubectl-v1372-helm-v431-and-uv-0136-in-the-crescent-workspace-2026-10-29). | PENDING | | H54a | On the Xeon 6741P, Gemma 4 26B-A4B (QAT weights) single-stream: the best optimized CPU arm, chosen by the pre-registered source and speed rule, decodes at N=1 at least 3x faster than EAT stock B1g (vLLM CPU v0.24.0 bf16, `xeon-values.yaml`'s Gemma entry on the QAT bf16 weights), with a 50-item quality spot-check no more than 10 points below B1g's. A microbench, not an agent claim. | Written 2026-10-20, before any H54 measurement. N=1 median decode tok/s: B1g in [4, 15] (0.6); Lg (llama.cpp CPU, Google QAT Q4_0) in [25, 70] (0.6); A1g (k3d-gpu auto, OVMS CPU, QAT-matched IR) in [15, 60] (0.5), A1g/A1o in [0.85, 1.02] (0.6); Vq (vLLM CPU v0.32.0 bf16) in [6, 18] (0.5), Vq-mtp/Vq in [1.4, 2.5] (0.5); Lg-mtp/Lg in [1.2, 2.2] (0.4); Sg (SGLang CPU AMX) deliverable (0.6). The chosen arm is Lg or Lg-mtp (0.55). B3g not deliverable (0.8). SUPPORTED (0.65), INCONCLUSIVE (0.25), REFUTED (0.1). Not gated: TTFT on a 1,000-token prompt, aggregate tok/s at N=4 and 16 (A1g N=16/N=1 <= 1.5, 0.5), J/token. | [H54 pre-registration](#h54-pre-registration-gemma-4-26b-a4b-qat-and-qwen38-microbenches-on-crescent-2026-10-20): `bench/llm_load.py` per arm (N=1: 8 requests, N=4: 16, N=16: 32, 256 tokens, greedy, `ignore_eos`), `bench/energy.sh` RAPL per phase, GSM8K items 0-49 graded by exact match, one arm serving at a time, at most 3 holds of at most 40 min. Statistic: ratio of the chosen arm's median N=1 decode to B1g's, 95% bootstrap CI over requests. SUPPORTED: CI lower bound >= 3.0 and the quality gate passed. REFUTED: CI upper bound < 3.0, or every arm with a point ratio >= 3.0 fails the quality gate. Otherwise INCONCLUSIVE (also: B1g not deliverable, or a void phase not re-run). Run 2026-10-21, 3 holds ([H54a run](#h54a-run-xeon-6741p-2026-10-21)): stock B1g stalled in both its holds (vLLM v0.24.0 `EngineDeadError` mid-decode, cause open; no scored phase, 0/50 GSM8K, all errored), so `h54.py score-a` has no base and no chosen arm. Measured: Lg 48.74, Lg-mtp 54.52, A1g 23.25, A1o 21.73 tok/s N=1 median, GSM8K 50, 50, 49, 49 of 50; each scored phase exists only under amendment R4 (its own foreign CPU 4.4-4.8 CPU-s/s). Vq, Vq-mtp, Vi, Sg, B3g not run (cap). Descriptive, not scored: Lg-mtp over B1g's 2 hold-2 rates 3.02 ~~(CI 2.90-3.06)~~ (descriptive only: n=2 base rates from an engine that died minutes later, no CI; correction 2026-10-28, [H54a run](#h54a-run-xeon-6741p-2026-10-21)). | INCONCLUSIVE | | H54q | QAT matters at int4 on this model: on GSM8K items 0-199, the QAT-matched OpenVINO IR OVQ (INT4_SYM, group 32, built from Google's QAT bf16 weights) is non-inferior to the bf16 QAT reference R within 5 points, and the published base-model IR OVB (`OpenVINO/gemma-4-26b-a4b-it-int4-ov`; ~~INT4_ASYM group 128~~ AWQ INT4 asymmetric group 64, router layers unquantized, per its `openvino_config.json`: [note N1](#h54-note-n1-what-ovb-is-2026-10-21)) scores below OVQ. | Written 2026-10-20, before any measurement. R accuracy in [0.85, 0.96] (0.6). OVQ - R in [-0.03, +0.02] (0.6). GGUF Q4_0 (Lg) - R in [-0.03, +0.02] (0.6). OVB - R in [-0.07, 0] (0.5). The OVB < OVQ part resolves on 200 items (0.2). INCONCLUSIVE (0.65), SUPPORTED (0.2), REFUTED (0.15). | As H54a's quality protocol on items 0-199 of `openai/gsm8k` `main` test @ `740312ad`; items 0-49 come from H54a's runs of the same arm configs; ~~at most 1 hold of at most 40 min for the rest~~ the rest in at most 3 holds of at most 40 min, R answering all 200 items because H54a never ran its config ([amendment Q1](#h54-amendment-q1-h54qs-reference-arm-hold-cap-and-order-2026-10-29)). Paired bootstrap 95% CI over items (10,000 resamples, seed 1). SUPPORTED: CI lower bound of OVQ - R >= -0.05 and CI upper bound of OVB - OVQ < 0. REFUTED: CI upper bound of OVQ - R < -0.05, or CI lower bound of OVB - OVQ > 0. Otherwise INCONCLUSIVE. | PENDING | | H54b | On Arc Pro B70 with Qwen3.8-27B single-stream: the best optimized arm on ONE card (k3d-gpu auto OVMS int4, Intel's llm-scaler vLLM, or llama.cpp SYCL, with and without each one's documented MTP) decodes at N=1 at least 2x faster than EAT #21's XPU path B2q on TWO cards (vLLM XPU v0.28.0, the `xpu-qwen36-27b` profile's flags on the Qwen3.8-27B weights, TP 2). | Written 2026-10-20, before any measurement. N=1 median decode tok/s: X1q (OVMS GPU int4) in [15, 28] (0.5), X1q-mtp/X1q in [1.3, 2.2] (0.4); Sq (llm-scaler 1.5, online FP8, TP 1) deliverable (0.5), in [10, 18]; Sq-mtp/Sq in [1.4, 2.3] (0.5); Lsq (llama.cpp SYCL, Intel Q4_K_M) in [12, 24] (0.5); B2q deliverable at TP 2 (0.4), in [5, 12] (0.5). Chosen/B2q >= 2 (0.6). Cards free for a window (0.5). INCONCLUSIVE (0.6), SUPPORTED (0.3), REFUTED (0.1). | H54a's protocol on the B70s, cards only when `~/k3dgpu-bench/gpu-state.txt` lists them free; at most 2 holds of at most 40 min. Same statistic and quality gate (against B2q, or the official-weights arm Sq when B2q is not deliverable, for the gate only). SUPPORTED: CI lower bound of chosen one-card arm / B2q >= 2.0 and the gate passed. REFUTED: CI upper bound < 2.0. Otherwise INCONCLUSIVE (also: B2q not deliverable, or no free card before the campaign ends). Gemma 4 QAT on one B70 (X1g, X1o, B2g, Sgg) is described if hold time remains, never gated. Run 2026-10-21, 2 holds ([H54b and H54c run](#h54b-and-h54c-run-arc-pro-b70-2026-10-21)): INCONCLUSIVE, not delivered as pre-registered: the pinned "one card" pods saw all 4 cards. B2q, Sq, Sq-mtp (Qwen bf16, 80.6 GB with its guard) and Rq were not deliverable (disk); X1q and X1q-mtp failed on two `h54b.sh` runner bugs before any engine started; Lsq ran, but its pod listed every render node and llama.cpp split the model over all four B70s, two of them serving the live `qwen38-a-hr`, so no one-card arm exists. Lsq-mtp was not run: its GGUF carries the MTP (`nextn`) tensors, but G1's runner has no such arm; nor were the described-only Gemma arms. `score.py`: "no gate reference (B2q and Sq missing)". Descriptive only, a 4-card run and not a one-card or B70 result: Lsq 27.3 tok/s N=1 median, GSM8K 48/50. | INCONCLUSIVE | @@ -4322,6 +4322,119 @@ interval), REFUTED (0.3), SUPPORTED (0.1). smoke, then holds 1-6 (and 7-8 if needed), then serving restored and verified. The window is at most eight 40-min holds with both pairs stopped; `gpu-state.txt` says so throughout. +### H53 amendment R1: the solo runner fix, setup, wiring smokes and the GPU window (2026-10-30) + +Recorded 2026-10-30 (UTC; 2026-10-29 in California), before any H53 hold or smoke. Mechanics +only. No hypothesis, arm, model, sampling, prediction, statistic, gate, threshold, seed or order +changes, and no cap rises. Like E1 to E4, it is pre-registered by its push time. + +State on crescent at 00:01Z: the only H53 record is `results/H53/prehold/fetch.log` (item 8). +`runs/campaign.log` and `runs/holds.log` have no H53 entry, no H53 process, cluster or lock +exists, the bench lock is free and both SGLang pairs serve. + +Disclosed: it is written with H50's, H50o's, H51's and H52b's results known. None of them runs +on a B70 or serves Qwen3.8-27B. + +1. **Deploy commit.** E2 item 2 runs each hold from `origin/main` at or after E2's merge, and E4 + item 2 runs E4's setup from a commit at or after E4's merge. H53 runs from one pushed commit on + branch `claude/h53-run`: `origin/main` `74ff63ef` (k3d-gpu 0.1.80, after E4's merge in + `012b4e95`) plus this amendment, item 2's fix, item 3's guard, item 5's futility helper and the + window script of item 5. It is checked out clean and detached at `~/k3dgpu-bench/k3d-gpu-h53`, + so a redeploy of the shared checkout cannot move H53's runner between holds. Its commit goes + into every hold's `commit.txt` and the run section. AG is `agent manifest` at this commit. +2. **The fix: `stage_path` was undefined on the solo path.** Since #157, `live.sh` defined + `stage_path` only inside its pairs branch, and the solo branch runs before it. AG's + `stage_path "${TGT_DIR}" "${TGT}" 5000` would have failed with "command not found". `live.sh` + runs without `-e`, so it would have gone on and deployed OVMS over an empty hostPath, and + hold 1 would have ended NOT DELIVERABLE because of the harness, not the arm. The function now + sits at top level, unchanged. The pairs path calls the same function. B2 mounts its weights + read-only (`LIVE_MOUNT_B=1`) and never called it. Regression test: + `H53Test.test_stage_path_is_defined_before_the_solo_path_calls_it` in + `bench/eatflow/test_pairs.py`. It fails on `74ff63ef`. +3. **A guard for the smokes.** `live.sh`'s solo branch passes `LIVE_PAIRS_GUARD`, when set, as + `agentbench solo --guard-s`. The scored holds leave it unset (`solo-hold.sh` does not set it), + so their guard stays agentbench's own. A smoke sets 99999 s: no task starts, and only the + unscored warm-up runs. +4. **Wiring smokes, inside the window, before hold 1.** No H53 arm has run on k3d. The + pre-registration's B2 boot-ladder smoke runs, and so does one for AG. Each is + `prehold.sh smoke [RUNGS]`: `hold.sh H53-smoke- 2400` running `live.sh H53` with + `solo-hold.sh`'s environment, except `LIVE_PAIRS_GUARD=99999`, `LIVE_HOLD` 1 for AG and 2 for + B2, and output under `~/k3dgpu-bench/smoke/H53//`. Each is discarded, disclosed in the + run section, and is not one of the eight holds. + - AG smoke (`H53-smoke-AG`). Pass: OVMS is ready on the GPU (`--target_device=GPU` in + `agent.yaml`), its pod holds exactly one free card (`live.sh`'s own check), and the warm-up + answers. If it fails for a reason the harness or k3d causes, no hold runs until a further + dated amendment fixes it. If OVMS itself does not load or answer, nothing changes, and + hold 1 runs as registered. + - B2 boot ladder (`H53-smoke-B2-a`, then one smoke per further rung, named by its letters, + such as `-b` or `-bd`). This is the pre-registration's ladder as written. Rung (a) is stock + with item 1's mounts dropped. A rung's smoke runs only when the boot before it fails with + that rung's error, and every rung taken is kept for every B2 hold (`LIVE_B2_LADDER`, the + `RUNGS` argument of `holds`). Pass: the server is ready, its pod holds the four free cards, + and the warm-up answers. If B2 does not answer after (d), H53 is INCONCLUSIVE (not + deliverable), no hold runs, and the failure is the finding. + - Early stop. A smoke may stop once its server is ready and the warm-up has answered, or once + its server has failed at least twice with the same error. Before stopping it, + `prehold.sh podlog ` saves the pods, events and server logs (current and previous). + Then a TERM goes to that smoke's `hold.sh`, whose cleanup deletes the cluster. +5. **The window** (`bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh`), in this order: + `setup` (item 7), `verify` (item 8) and `frontdoor pre` before the window; `stop`; the smokes; + `holds 1 6 RUNGS`; `start`. + - `stop` saves `docker inspect` of both pairs into `prehold/` first. Then it runs + `gpu-serving.sh stop` for `qwen38-a-hr` and `qwen38-b-hr` (`docker stop`, never `rm`; it + refuses while aiperf runs). + - Before every smoke and hold, the window must be open, both pairs must be stopped, and every + card must hold under 2 GiB of VRAM (Prometheus `xe_vram_used_bytes`, polled for up to + 2 min). Else the step does not run. The serving pairs hold about 16 GiB of each card. + - `holds FIRST LAST RUNGS` runs the scored holds in order through `solo-hold.sh N ARM`, with + `LIVE_GPU_FREE` set to all four cards. After hold 6, holds 7 and 8 run only when `agentbench + solo-report` (on a copy of the solo directory) counts fewer than three blocks. This is the + pre-registered rule. Then it restores serving. + - A hold that does not exit 0 counts as used, and `holds` stops there with the window open. + The hold's records are read before the next hold. A failure the harness or k3d causes takes a + further dated amendment before any further hold. A failure of the arm itself (it does not + load, does not answer, or stops answering) is that arm's result, and the holds go on in + order from the next one. + - Futility stop. After every hold, `solo.alive_blocks` lists the blocks (1 to 4) that can + still count: a block is out when one of its holds has run with fewer than 8 complete + attempts, or when both have run and it has fewer than 8 complete pairs. When fewer than + three remain, no further hold runs and serving is restored. The verdict is then INCONCLUSIVE + by the pre-registered rule (fewer than three blocks count), whatever later holds would show. + The rule reads only block counts, never d or a success rate, and it shortens the outage of + the serving pairs. Tests: `AliveBlocksTest` in `bench/eatflow/test_solo.py`. + - `start` (and the end of `holds`) runs `gpu-serving.sh start` for both pairs. That waits for + `/health`, `/v1/models` and a 16-token chat at each pair's own port. It then compares each + pair's image, command, env, devices, mounts and ports with the saved inspect, and runs E2 + item 7's front-door check through `:8100`. The front-door check gates nothing. + - A deadman runs from `stop`. If no window step is alive and none has run for 45 min, it + takes the bench lock and restores both pairs, so a dead session cannot leave serving down. + It stays one window (E2 item 4): smokes and holds 1-8 run with both pairs stopped, and + serving comes back after the last. +6. **Placement records.** `live.sh` already stops a hold whose pod holds a card outside + `LIVE_GPU_FREE` or the wrong number of cards. For every hold, the window script also saves, + from Prometheus, each card's VRAM and engine use over the hold (`hold/xe-prom.json`). These + are reported and gate nothing. +7. **Tools (crescent amendment E4).** E4 merged before this push, and H53 is one of its rows. So + `prehold.sh setup` runs `bench/eatflow/crescent/setup.sh` once from H53's checkout, under the + bench lock, before the first smoke. Its output, the checksum lines and the four versions, goes + to `prehold/setup.txt`. Every smoke and hold then runs on k3d v5.9.0, kubectl v1.37.2, helm + v4.3.1 and uv 0.13.6 in `~/k3dgpu-bench/bin`. E4's render check found B2's staged objects + byte-identical under helm v4.1.1 and v4.3.1. setup.sh also re-extracts `eat-pr21` from + `c86c42b`, the tree it already holds, and re-checks the banking venv's packages. +8. **Models.** `models.sh qwen38` ran under the bench lock from 21:44Z to 21:55Z on 2026-10-29, + from the main-tracking checkout at `6d5cab69`, before this amendment existed. `models.sh` and + `listing.sh` are unchanged from `6d5cab69` to the deploy commit. It downloaded both models at + their pinned revisions, checked every LFS file against the Hub's sha256 (58 files, all `ok`, + rc 0) and wrote B2's listing, which `live.sh` checks at every B2 hold. Its log is + `prehold/fetch.log`, the record `prehold.sh fetch` writes. It measures nothing. Before the + window, `prehold.sh verify` checks both models against the Hub again, and B2's listing against + setup's (`prehold/verify.txt`). +9. **Records.** The results come back under the pre-registration's date, + `bench/eatflow/results/2026-10-20/H53/`. `solo-hold.sh` writes `pairs/` and `hold/`, and + `hold.sh` writes `runs/H53-hold-/`. The run also keeps `prehold/`, `window/` and + `smoke//`. The score is `agentbench solo-report bench/eatflow/results/2026-10-20/H53/pairs + --hyp H53 --arm-a AG --arm-b B2`, at the deploy commit. + ### H54 pre-registration: Gemma 4 26B-A4B QAT and Qwen3.8 microbenches on crescent (2026-10-20) Written 2026-10-20, before any H54a, H54q, H54b, H54c or H54d measurement. Nothing here is a diff --git a/bench/eatflow/live.sh b/bench/eatflow/live.sh index 28a1dba7..1688771e 100755 --- a/bench/eatflow/live.sh +++ b/bench/eatflow/live.sh @@ -280,6 +280,19 @@ stage() { # stage docker exec "${n}" chown -R "$3:$3" /var/apb-models done } +# stage_path : copy into every node, chown that +# path only. Top level, so the solo (H53) and pairs paths both have it (H53 amendment R1). +stage_path() { + local n + for n in $(nodes); do + docker exec "${n}" mkdir -p "/var/apb-models/$2" /var/apb-models/.ov_cache + docker cp -L "${M}/$1/." "${n}:/var/apb-models/$2/" + docker exec "${n}" rm -rf "/var/apb-models/$2/.cache" "/var/apb-models/$2/.git" + docker exec "${n}" chown -R "$3:$3" "/var/apb-models/$2" + docker exec "${n}" chown 5000:5000 /var/apb-models/.ov_cache + docker exec "${n}" chmod 1777 /var/apb-models + done +} # cpusets: H56's cpuset.txt, " " lines that crescent/h56.py checks. cpusets() { local n c @@ -316,6 +329,8 @@ if [ "${MODE}" = solo ]; then # AG (k3d-gpu's auto device and draft: OVMS on one card) or B2 (EAT PR #21's vLLM XPU path, # its xpu-qwen36-27b profile on these weights, LIVE_B2_TP cards). LIVE_GPU_FREE: the PCI # addresses of the cards the hold may use; every other card is held by a placeholder pod. + # LIVE_PAIRS_GUARD: the task-start guard, s (unset: agentbench solo's own; a wiring smoke + # sets it past the hold, so only the warm-up runs; H53 amendment R1). PD=${LIVE_PAIRS_DIR:?} HOLD=${LIVE_HOLD:?} SOLO=${LIVE_SOLO:?LIVE_SOLO is AG or B2} read -r FAMILY RULE <<< "$(python3 "${HERE}/pairs.py" design "${ARM}")" [ "${ARM}" = H53 ] && [ "${RULE:-}" = L3 ] || { echo "solo: ARM must be H53 (a rule-L3 row)"; exit 2; } @@ -428,7 +443,8 @@ if [ "${MODE}" = solo ]; then ${AB} solo --hyp "${ARM}" --hold "${HOLD}" --arm "${SOLO}" --out "${PD}" --deadline "${deadline}" --tasks B24 \ --agent http://127.0.0.1:8090 --upstream "http://127.0.0.1:${P_LLM}" --proxy-port "${P_PROXY}" \ --restart "${OUT}/agent.sh" --pod "${cg}" --mcp-url "http://127.0.0.1:${P_MCP}/mcp" --reset "${OUT}/reset-mcp.sh" \ - --sampling "${SAMPLING}" --baseline "${PD}/baseline-hold${HOLD}.jsonl" --level "${LEVEL}" 2>&1 | tee -a "${OUT}/live.log" + --sampling "${SAMPLING}" --baseline "${PD}/baseline-hold${HOLD}.jsonl" --level "${LEVEL}" \ + ${LIVE_PAIRS_GUARD:+--guard-s "${LIVE_PAIRS_GUARD}"} 2>&1 | tee -a "${OUT}/live.log" stamp finished exit 0 fi @@ -462,17 +478,6 @@ if [ "${MODE}" = pairs ]; then B_NS=default B_DEP=${B_REL} B_SVC=${B_REL}-service B_SPORT=80 B_API=/v1 [ "${DESIGN}" = H50o ] && B_DEP=ovms-model-server B_SVC=ovms-model-server B_SPORT=8080 B_API=/v3 DRAFT_DIR=OpenVINO-Qwen3-0.6B-int4-ov # where OVMS --pull puts the draft, inside the target dir - stage_path() { # stage_path : chown that path only - local n - for n in $(nodes); do - docker exec "${n}" mkdir -p "/var/apb-models/$2" /var/apb-models/.ov_cache - docker cp -L "${M}/$1/." "${n}:/var/apb-models/$2/" - docker exec "${n}" rm -rf "/var/apb-models/$2/.cache" "/var/apb-models/$2/.git" - docker exec "${n}" chown -R "$3:$3" "/var/apb-models/$2" - docker exec "${n}" chown 5000:5000 /var/apb-models/.ov_cache - docker exec "${n}" chmod 1777 /var/apb-models - done - } cluster_up stage_path "${TGT_DIR}" "${TGT}" 5000 [ "${A_DRAFT}" = off ] || stage_path Qwen3-0.6B-int4-ov "${TGT}/${DRAFT_DIR}" 5000 diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh b/bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh new file mode 100755 index 00000000..7a51ba8a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh @@ -0,0 +1,304 @@ +#!/usr/bin/env bash +# prehold.sh STEP: H53's window steps on crescent (EXPERIMENTS.md, H53 amendment R1), run from the +# H53 checkout ~/k3dgpu-bench/k3d-gpu-h53 at the deploy commit. +# fetch models.sh qwen38 under the bench lock: both models at their pinned revisions, +# every LFS file against the Hub's sha256, B2's listing (prehold/fetch.log) +# setup crescent amendment E4: setup.sh under the bench lock (prehold/setup.txt) +# verify both models again against the Hub's sha256, and B2's listing against setup's +# (prehold/verify.txt) +# frontdoor TAG E2 item 7: a 16-token chat through Dynamo :8100 per serving pair, passing when it +# answers with content and that pair's SGLang num_requests_total rises (180 s each) +# stop open the window: `docker inspect` of both pairs into prehold/, then +# gpu-serving.sh stop for each (never rm), the cards' VRAM, window/open and the deadman +# smoke AG|B2 [RUNGS] +# a wiring smoke: solo-hold.sh's environment, except LIVE_PAIRS_GUARD=99999 (only the +# warm-up runs), LIVE_HOLD 1 (AG) or 2 (B2), B2's ladder RUNGS, output under smoke/H53/ +# podlog NAME a running smoke's pods, events and server logs (current and previous) into its dir +# holds FIRST LAST [RUNGS] +# the scored holds FIRST..LAST through solo-hold.sh, each on the four free cards and +# recorded from Prometheus; stops at the first hold that does not exit 0 (the window +# stays open), or once fewer than three blocks can still count (then restores); +# after hold 6, holds 7 and 8 run only when fewer than three blocks count; then `start` +# start close the window: gpu-serving.sh start for both pairs, the inspect comparison and +# frontdoor post +# status refresh the heartbeat; print the window's last events and the cards' VRAM +# deadman started by stop: restores both pairs once no window step is alive and none has +# run for 45 min (window/heartbeat), so a dead session cannot leave serving down +set -uo pipefail +B=${HOME}/k3dgpu-bench +REPO=${B}/k3d-gpu-h53 +C=${REPO}/bench/eatflow +R=${B}/results/H53 +O=${R}/prehold +W=${R}/window +S=${B}/smoke/H53 +mkdir -p "${O}" "${W}" +NODE_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c +PAIRS="qwen38-a-hr qwen38-b-hr" +# All four Arc Pro B70s: both pairs are stopped for the window (H53 pre-registration). +FREE="0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0" +PROM=http://127.0.0.1:9090 +STALE_S=2700 +FREE_GIB=2 +head_of() { (cd "${REPO}" && git rev-parse HEAD); } +ev() { echo "$(date -u +%FT%TZ) $*" | tee -a "${W}/events.log"; } +beat() { touch "${W}/heartbeat"; } +running() { docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null; } +alive() { [ -f "${W}/active.pid" ] && kill -0 "$(cat "${W}/active.pid")" 2>/dev/null; } +lock() { # lock: the bench lock on fd 9, as hold.sh takes it + exec 9> "${HOME}/k3dgpu-bench.lock" + flock -w 7200 9 || { echo "lock not taken in 7200 s"; exit 4; } +} +# vram: one " " line per card from Prometheus (xe-exporter's xe_vram_used_bytes). +vram() { + curl -s -m 10 "${PROM}/api/v1/query?query=xe_vram_used_bytes" | python3 -c ' +import json, sys +for r in sorted(json.load(sys.stdin)["data"]["result"], key=lambda r: r["metric"]["bdf"]): + print(r["metric"]["bdf"], round(int(r["value"][1]) / 2**30, 2))' +} +# ready_cards: the window is open, both pairs are stopped and every card holds under FREE_GIB. +ready_cards() { + local c v + [ -e "${W}/open" ] || { ev "window not open"; return 1; } + for c in ${PAIRS}; do + [ "$(running "${c}")" = false ] || { ev "${c} is running"; return 1; } + done + v=$(vram) || return 1 + [ "$(wc -l <<< "${v}")" -eq 4 ] || { ev "Prometheus lists $(wc -l <<< "${v}") cards, not 4"; return 1; } + if awk -v g="${FREE_GIB}" '$2 >= g {bad=1} END {exit !bad}' <<< "${v}"; then + ev "a card holds ${FREE_GIB} GiB or more: $(tr '\n' ';' <<< "${v}")"; return 1 + fi + ev "cards free: $(tr '\n' ';' <<< "${v}")" +} +# wait_cards: ready_cards, polled for up to 2 min (Prometheus scrapes every 15 s). +wait_cards() { + local _ + for _ in $(seq 24); do ready_cards > /dev/null 2>&1 && break; sleep 5; done + ready_cards +} +# prom T0 T1 OUT: per-card VRAM and engine utilisation over [T0, T1] (epoch s), 15 s steps. +prom() { + python3 - "$1" "$2" "$3" "${PROM}" <<'PY' +import json, sys, urllib.parse, urllib.request +t0, t1, out, prom = float(sys.argv[1]), float(sys.argv[2]), sys.argv[3], sys.argv[4] +res = {"t0": t0, "t1": t1} +for q in ("xe_vram_used_bytes", "xe_engine_utilization_percent"): + u = f"{prom}/api/v1/query_range?" + urllib.parse.urlencode({"query": q, "start": t0, "end": t1, "step": 15}) + with urllib.request.urlopen(u, timeout=30) as r: + res[q] = json.load(r)["data"]["result"] +peak = {} +for s in res["xe_vram_used_bytes"]: + peak[s["metric"]["bdf"]] = round(max(int(v) for _, v in s["values"]) / 2**30, 2) +res["vram_peak_gib"] = peak +busy = {} +for s in res["xe_engine_utilization_percent"]: + b = s["metric"].get("bdf", "?") + busy[b] = max(busy.get(b, 0.0), max(float(v) for _, v in s["values"])) +res["engine_peak_pct"] = {k: round(v, 1) for k, v in sorted(busy.items())} +with open(out, "w") as f: + json.dump(res, f) +print("vram peak GiB", peak, "engine peak %", res["engine_peak_pct"]) +PY +} +frontdoor() { + local f=${O}/frontdoor-${1:-check}.txt P b end ok c + : > "${f}" + n() { curl -s -m 8 "http://127.0.0.1:$1/metrics" | awk '/^sglang:num_requests_total/{s+=$2} END{print s+0}'; } + for P in 30000 30001; do + b=$(n "${P}") end=$(( SECONDS + 180 )) ok=0 c="" + until [ "${SECONDS}" -ge "${end}" ]; do + c=$(curl -s -m 60 http://127.0.0.1:8100/v1/chat/completions -H 'Content-Type: application/json' \ + -d '{"model":"Qwen3.8-27B","messages":[{"role":"user","content":"Say OK"}],"max_tokens":16,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}' \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["choices"][0]["message"]["content"].strip())' 2>/dev/null) + if [ -n "${c}" ] && [ "$(n "${P}")" -gt "${b}" ]; then ok=1; break; fi + sleep 5 + done + echo "$(date -u +%FT%TZ) :${P} pass=${ok} before=${b} after=$(n "${P}") content=${c:-none}" | tee -a "${f}" + done +} +# The parts of `docker inspect` a restart must keep: image, command, env, devices, mounts, ports. +inspect_key() { + docker inspect "$1" | python3 -c ' +import json, sys +d = json.load(sys.stdin)[0] +h = d["HostConfig"] +print(json.dumps({"image": d["Image"], "path": d["Path"], "args": d["Args"], "env": sorted(d["Config"]["Env"]), + "devices": h.get("Devices"), "binds": h.get("Binds"), "network": h.get("NetworkMode"), + "ports": h.get("PortBindings"), "restart": h.get("RestartPolicy")}, sort_keys=True))' +} +restore() { + local c p pids=() rc=0 + rm -f "${W}/open" + for c in ${PAIRS}; do + if [ "$(running "${c}")" = true ]; then ev "${c} already running"; continue; fi + bash "${C}/crescent/gpu-serving.sh" start "${c}" >> "${W}/restore.log" 2>&1 & + pids+=($!) + done + for p in "${pids[@]}"; do wait "${p}" || rc=1; done + for c in ${PAIRS}; do + ev "${c} running=$(running "${c}"); $(bash "${C}/crescent/gpu-serving.sh" check "${c}" 2>&1 | head -1)" + inspect_key "${c}" > "${O}/inspect-${c}.after.key.json" + if cmp -s "${O}/inspect-${c}.before.key.json" "${O}/inspect-${c}.after.key.json"; then + ev "${c}: image, command, env, devices, mounts and ports as before the stop" + else + ev "${c}: inspect DIFFERS from before the stop"; rc=1 + fi + done + frontdoor post + ev "window closed (restore rc=${rc})" + return "${rc}" +} +blocks() { # the number of blocks that count so far (solo-report's block_level.n_blocks, on a copy) + local t + t=$(mktemp -d) + cp "${R}/pairs/registration.json" "${R}/pairs/attempts.jsonl" "${R}/pairs/solo.jsonl" "${t}/" 2>/dev/null + python3 "${C}/agentbench.py" solo-report "${t}" --hyp H53 --arm-a AG --arm-b B2 > /dev/null 2>&1 + python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["block_level"]["n_blocks"])' "${t}/analysis.json" 2>/dev/null || echo 0 + rm -rf "${t}" +} +alive_blocks() { # the blocks that can still count (solo.alive_blocks); a hold has run once hold.sh wrote its rc + local ran="" n + for n in 1 2 3 4 5 6 7 8; do + compgen -G "${B}/runs/H53-hold${n}-*/rc" > /dev/null && ran="${ran} ${n}" + done + python3 - "${C}" "${R}/pairs/solo.jsonl" "${ran}" <<'PY' +import os, sys +sys.path.insert(0, sys.argv[1]) +import pairs as PR, solo +rows = PR.read_jsonl(sys.argv[2]) if os.path.exists(sys.argv[2]) else [] +print(" ".join(map(str, solo.alive_blocks(rows, {int(x) for x in sys.argv[3].split()})))) +PY +} +one_hold() { # one_hold N RUNGS + local n=$1 arm t0 t1 rc + arm=$(python3 "${C}/pairs.py" arm H53 "${n}") || return 2 + beat + wait_cards || { ev "hold ${n}: cards not free, no hold"; return 6; } + t0=$(date +%s) + ev "hold ${n} (${arm}) start" + LIVE_GPU_FREE="${FREE}" LIVE_B2_LADDER="$2" bash "${C}/crescent/solo-hold.sh" "${n}" "${arm}" + rc=$? + t1=$(date +%s) + beat + mkdir -p "${R}/hold${n}" + prom "${t0}" "${t1}" "${R}/hold${n}/xe-prom.json" >> "${W}/events.log" 2>&1 + ev "hold ${n} (${arm}) rc=${rc} ($(( t1 - t0 )) s); $(grep -h 'complete attempt\|ABORTED\|NOT DELIVERABLE' "${R}/hold${n}/live.log" 2>/dev/null | tail -1)" + return "${rc}" +} +case ${1:?usage: prehold.sh fetch|setup|verify|frontdoor|stop|smoke|podlog|holds|start|status|deadman} in +fetch) + lock + { echo "lock-taken $(date -u +%FT%TZ)"; head_of + HF_HUB_DISABLE_PROGRESS_BARS=1 bash "${C}/crescent/models.sh" qwen38 + echo "rc=$? $(date -u +%FT%TZ)"; } > "${O}/fetch.log" 2>&1 ;; +setup) + lock + { echo "lock-taken $(date -u +%FT%TZ)"; head_of + bash "${C}/crescent/setup.sh" + echo "rc=$? $(date -u +%FT%TZ)" + (cd "${B}/bin" && sha256sum k3d kubectl helm uv); } > "${O}/setup.txt" 2>&1 + tail -12 "${O}/setup.txt" ;; +verify) + lock + { echo "lock-taken $(date -u +%FT%TZ)"; head_of + bash "${C}/crescent/models.sh" verify "${B}/models/Qwen3.8-27B" Qwen/Qwen3.8-27B 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 + echo "Qwen3.8-27B rc=$?" + bash "${C}/crescent/models.sh" verify "${B}/models/Qwen3.8-27B-int4-ov" OpenVINO/Qwen3.8-27B-int4-ov 7b20ea8c7df7223a85ff27bab5c473f0cb038943 + echo "Qwen3.8-27B-int4-ov rc=$?" + bash "${C}/crescent/listing.sh" "${B}/models/Qwen3.8-27B" | cmp -s - "${B}/models/Qwen3.8-27B.listing" + echo "listing equal to setup's: rc=$?"; date -u +%FT%TZ; } > "${O}/verify.txt" 2>&1 + grep -E 'rc=|BAD|MISSING' "${O}/verify.txt" ;; +frontdoor) frontdoor "${2:-check}" ;; +stop) + [ ! -e "${W}/open" ] || { ev "window already open"; exit 1; } + for c in ${PAIRS}; do + docker inspect "${c}" > "${O}/inspect-${c}.before.json" || exit 1 + inspect_key "${c}" > "${O}/inspect-${c}.before.key.json" || exit 1 + done + ev "VRAM before the stop: $(vram | tr '\n' ';')" + date -u +%FT%TZ > "${W}/open" + beat + for c in ${PAIRS}; do + if ! bash "${C}/crescent/gpu-serving.sh" stop "${c}" >> "${W}/stop.log" 2>&1; then + ev "${c} NOT stopped (see stop.log): restoring"; restore; exit 1 + fi + ev "${c} stopped" + done + setsid nohup bash "$0" deadman > "${W}/deadman.log" 2>&1 < /dev/null & + echo $! > "${W}/deadman.pid" + ev "window open; deadman pid $(cat "${W}/deadman.pid")" + wait_cards ;; +smoke) + ARM=${2:?usage: prehold.sh smoke AG|B2 [RUNGS]} RUNGS=${3:-} + case ${ARM} in AG) H=1 ;; B2) H=2 ;; *) echo "smoke: ARM must be AG or B2"; exit 2 ;; esac + name=H53-smoke-${ARM}${RUNGS:+-${RUNGS}} + [ "${ARM}" = AG ] || name=H53-smoke-B2-${RUNGS:-a} + beat + wait_cards || exit 6 + mkdir -p "${S}/${name}/pairs" + echo $$ > "${W}/active.pid" + ev "smoke ${name} start" + exec "${C}/crescent/hold.sh" "${name}" 2400 env K3D_GPU_IMAGE="${NODE_IMAGE}" K3D_GPU_K3S_ARGS="--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1%" \ + LIVE_MODE=solo LIVE_SOLO="${ARM}" LIVE_CAP=2370 LIVE_HOLD="${H}" LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR="${S}/${name}/pairs" \ + LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE="${FREE}" LIVE_B2_TP=4 LIVE_B2_LADDER="${RUNGS}" \ + LIVE_EAT_XPU_CHART="${B}/eat-pr21/core/helm-charts/vllm" LIVE_MODELS="${B}/models" LIVE_BLUEPRINT="${B}/eab" \ + LIVE_PY="${B}/mafvenv/bin/python" PATH="${B}/bin:${PATH}" \ + bash "${C}/live.sh" H53 "${S}/${name}/hold" ;; +podlog) + d=${S}/${2:?usage: prehold.sh podlog NAME}/hold + export KUBECONFIG=${d}/kubeconfig PATH=${B}/bin:${PATH} + beat + kubectl get pods -A -o wide > "${d}/pods.txt" 2>&1 + kubectl get events -A --sort-by=.lastTimestamp > "${d}/events.txt" 2>&1 + for ns in default k3d-gpu-agent; do + for p in $(kubectl -n "${ns}" get pods -o name 2>/dev/null | grep -v gpu-hold); do + kubectl -n "${ns}" describe "${p}" > "${d}/describe-${p##*/}.txt" 2>&1 + kubectl -n "${ns}" logs "${p}" --all-containers > "${d}/log-${p##*/}.txt" 2>&1 + kubectl -n "${ns}" logs "${p}" --all-containers --previous > "${d}/log-${p##*/}.previous.txt" 2>&1 + done + done + ls -l "${d}" ;; +holds) + first=${2:?usage: prehold.sh holds FIRST LAST [RUNGS]} last=${3:?} rungs=${4:-} + echo $$ > "${W}/active.pid" + ev "holds ${first}-${last} (ladder '${rungs}') at $(head_of)" + n=${first} + while [ "${n}" -le "${last}" ]; do + one_hold "${n}" "${rungs}" || { ev "hold ${n} did not exit 0: holds stop, the window stays open"; exit 1; } + a=$(alive_blocks) + ev "after hold ${n}: blocks that can still count: ${a:-none}" + if [ "$(wc -w <<< "${a}")" -lt 3 ]; then + ev "fewer than three blocks can still count: no further hold (futility, R1 item 5)"; break + fi + if [ "${n}" -eq 6 ] && [ "${last}" -eq 6 ]; then + k=$(blocks) + ev "after hold 6: ${k} block(s) count" + [ "${k}" -ge 3 ] || { last=8; ev "fewer than three blocks count: holds 7 and 8 run"; } + fi + n=$(( n + 1 )) + done + restore ;; +start) beat; restore ;; +status) + beat + tail -n "${2:-8}" "${W}/events.log" + vram | tr '\n' ';'; echo ;; +deadman) + while [ -e "${W}/open" ]; do + sleep 60 + [ -e "${W}/open" ] || break + alive && continue + age=$(( $(date +%s) - $(stat -c %Y "${W}/heartbeat" 2>/dev/null || echo 0) )) + [ "${age}" -ge "${STALE_S}" ] || continue + # No hold of any row runs while serving comes back: take the bench lock first. + exec 9> "${HOME}/k3dgpu-bench.lock" + flock -w 7200 9 || { ev "deadman: lock not taken"; continue; } + if [ -e "${W}/open" ] && ! alive; then + ev "deadman: no window step for ${age} s: restoring serving" + restore + fi + flock -u 9 + break + done ;; +*) echo "usage: prehold.sh fetch|setup|verify|frontdoor|stop|smoke|podlog|holds|start|status|deadman"; exit 2 ;; +esac diff --git a/bench/eatflow/solo.py b/bench/eatflow/solo.py index a7ddabcd..88e01df9 100644 --- a/bench/eatflow/solo.py +++ b/bench/eatflow/solo.py @@ -126,6 +126,32 @@ def block_pairs(attempts, solo_rows, a, b): MIN_BLOCKS = 3 +MAX_BLOCKS = 4 # holds 7 and 8 are the last block (no hold runs after eight) + + +def alive_blocks(solo_rows, ran, min_pairs=MIN_BLOCK_PAIRS, max_blocks=MAX_BLOCKS): + """H53 amendment R1's futility rule: the blocks (1..max_blocks) that can still count, given + solo.jsonl's rows and `ran`, the hold numbers that have run (a hold that delivered nothing has + run but has no rows). A block is out when a hold of it has run with fewer than min_pairs + complete attempts (its complete pairs can be no more), or when both of its holds have run and + it has fewer than min_pairs complete pairs. It reads only which blocks may count, never d.""" + status = {(r["hold"], r["j"]): r["status"] for r in solo_rows} + done = {} + for (h, _), s in status.items(): + done[h] = done.get(h, 0) + (s == "complete") + alive = [] + for k in range(1, max_blocks + 1): + h1, h2 = 2 * k - 1, 2 * k + if any(h in ran and done.get(h, 0) < min_pairs for h in (h1, h2)): + continue + if h1 in ran and h2 in ran: + js = {j for (h, j) in status if h == h1} + if sum(status.get((h1, j)) == "complete" and status.get((h2, j)) == "complete" for j in js) < min_pairs: + continue + alive.append(k) + return alive + + def block_verdict(res, a, b, threshold=0.30, min_rate=0.6): """The H53 verdict from pairs.analyze's result over block pairs (key 1000 * k + j). Per block k: d_k = mean over its matched pairs of ok(a) - ok(b). Over the n blocks with at least diff --git a/bench/eatflow/test_pairs.py b/bench/eatflow/test_pairs.py index 09a525cb..97e4713d 100644 --- a/bench/eatflow/test_pairs.py +++ b/bench/eatflow/test_pairs.py @@ -888,6 +888,68 @@ def test_live_sh_keeps_the_arm_off_the_serving_cards(self): self.assertLess(i_hold, i_apply) self.assertLess(i_check, i_run) + def test_stage_path_is_defined_before_the_solo_path_calls_it(self): + # H53 amendment R1: stage_path lived inside the pairs branch, so the solo AG path called an + # undefined function (bash went on: no -e) and OVMS would have served an empty hostPath. + live = self.live() + self.assertEqual(live.count("stage_path() {"), 1) + i_def = live.index("\nstage_path() {") # top level, not inside a branch + i_solo = live.index('if [ "${MODE}" = solo ]; then') + self.assertLess(i_def, i_solo) + blk = live[i_solo:live.index('if [ "${MODE}" = pairs ]; then')] + self.assertIn('stage_path "${TGT_DIR}" "${TGT}" 5000', blk) + # Run the definition as bash does, with stub nodes/docker: it copies into every node. + fn = live[i_def + 1:live.index("\n}\n", i_def) + 3] + tmp = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmp) + script = ("set -u\nM=/m\nnodes() { echo n1 n2; }\ndocker() { echo \"docker $*\" >> " + tmp + "/calls; }\n" + + fn + 'stage_path Qwen3.8-27B-int4-ov OpenVINO/Qwen3.8-27B-int4-ov 5000\n') + import subprocess + p = subprocess.run(["bash", "-c", script], capture_output=True, text=True) + self.assertEqual(p.returncode, 0, p.stderr) + with open(os.path.join(tmp, "calls")) as f: + calls = f.read() + for n in ("n1", "n2"): + self.assertIn(f"docker cp -L /m/Qwen3.8-27B-int4-ov/. {n}:/var/apb-models/OpenVINO/Qwen3.8-27B-int4-ov/", calls) + self.assertIn(f"docker exec {n} chown -R 5000:5000 /var/apb-models/OpenVINO/Qwen3.8-27B-int4-ov", calls) + + def test_a_smoke_can_set_the_solo_task_guard(self): + live = self.live() + blk = live[live.index('if [ "${MODE}" = solo ]; then'):live.index('if [ "${MODE}" = pairs ]; then')] + self.assertIn('${LIVE_PAIRS_GUARD:+--guard-s "${LIVE_PAIRS_GUARD}"}', blk) + with open(os.path.join(HERE, "crescent", "solo-hold.sh")) as f: + self.assertNotIn("LIVE_PAIRS_GUARD", f.read()) # holds keep agentbench's own guard + + def test_h53_window_steps(self): + p = os.path.join(HERE, "results", "2026-10-20", "H53", "prehold", "prehold.sh") + with open(p) as f: + s = f.read() + # The four cards of both pairs (pre-registration "Cards and serving"), stop and start only. + self.assertIn('FREE="0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0"', s) + self.assertIn('PAIRS="qwen38-a-hr qwen38-b-hr"', s) + self.assertNotIn("docker rm", s) + self.assertNotIn("docker stop", s) # only through gpu-serving.sh + i_inspect, i_stop = s.index('docker inspect "${c}" > "${O}/inspect-${c}.before.json"'), s.index("gpu-serving.sh\" stop") + self.assertLess(i_inspect, i_stop) + # A smoke runs the warm-up only; holds run solo-hold.sh with the registered arm. + self.assertIn("LIVE_PAIRS_GUARD=99999", s) + self.assertIn('bash "${C}/crescent/solo-hold.sh" "${n}" "${arm}"', s) + self.assertIn('arm=$(python3 "${C}/pairs.py" arm H53 "${n}")', s) + # Holds 7 and 8 only when fewer than three blocks count after hold 6. + self.assertIn('[ "${k}" -ge 3 ] || { last=8;', s) + # Every card must be free before a smoke or a hold (polled, Prometheus scrapes every 15 s). + self.assertEqual(s.count("wait_cards ||"), 2) + self.assertIn("FREE_GIB=2", s) + # R1 item 5's futility stop reads solo.alive_blocks after every hold. + i_hold, i_alive = s.index('one_hold "${n}" "${rungs}" ||'), s.index("a=$(alive_blocks)") + self.assertLess(i_hold, i_alive) + self.assertIn('[ "$(wc -w <<< "${a}")" -lt 3 ]', s) + self.assertIn("solo.alive_blocks(rows,", s) + # E4: setup.sh runs from this checkout under the bench lock. + blk = s[s.index("\nsetup)"):s.index("\nverify)")] + self.assertIn("lock", blk) + self.assertIn('bash "${C}/crescent/setup.sh"', blk) + class PrerunPinsTest(unittest.TestCase): """Crescent amendments E1 and E2 (EXPERIMENTS.md, 2026-10-29): every crescent agent-row diff --git a/bench/eatflow/test_solo.py b/bench/eatflow/test_solo.py index 8d4423cf..4cd97471 100644 --- a/bench/eatflow/test_solo.py +++ b/bench/eatflow/test_solo.py @@ -199,6 +199,32 @@ def test_a_short_block_is_not_counted(self): self.assertEqual((bv["n_blocks"], bv["verdict"]), (2, "INCONCLUSIVE")) +class AliveBlocksTest(unittest.TestCase): + """H53 amendment R1's futility rule: which blocks can still count.""" + + @staticmethod + def rows(hold, n, void=()): + return [{"hold": hold, "j": j, "status": "void" if j in void else "complete"} for j in range(n)] + + def test_nothing_run_leaves_every_block(self): + self.assertEqual(S.alive_blocks([], set()), [1, 2, 3, 4]) + + def test_a_hold_that_delivered_nothing_takes_its_block_out(self): + # Hold 1 ran (rc recorded) with no rows: block 1 cannot reach 8 pairs; 2-4 still can. + self.assertEqual(S.alive_blocks([], {1}), [2, 3, 4]) + self.assertEqual(S.alive_blocks(self.rows(2, 24), {1, 2}), [2, 3, 4]) + + def test_a_short_hold_and_a_short_block(self): + # Hold 3 ran 7 complete attempts: block 2 is out. Holds 5 and 6 ran 10 each but only + # 7 tasks overlap complete (hold 6 voided three): block 3 is out. + rows = self.rows(1, 12) + self.rows(2, 12) + self.rows(3, 7) + self.rows(5, 10) + self.rows(6, 10, void={0, 1, 2}) + self.assertEqual(S.alive_blocks(rows, {1, 2, 3, 5, 6}), [1, 4]) + + def test_two_blocks_out_leave_fewer_than_three(self): + rows = self.rows(2, 24) + self.rows(3, 24) + self.assertEqual(len(S.alive_blocks(rows, {1, 2, 3, 4})), 2) # holds 1 and 4 (AG) delivered nothing + + class RunSoloTest(unittest.TestCase): def setUp(self): self.tmp = tempfile.mkdtemp() From 2262dcb37fcd2441d2496fdf304f9927601ef76c Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Thu, 29 Oct 2026 17:45:20 -0700 Subject: [PATCH 2/2] ledger(H53): INCONCLUSIVE, B2 not deliverable on k3d-gpu; the run's records H53's window ran on crescent from 1f39488b (amendment R1). After E4's setup and a model re-verify, both serving pairs were stopped with docker stop at 00:15Z. The wiring smokes ran, and both pairs were restored at 00:37Z. - AG's smoke passed. agent manifest chose the GPU, and OVMS 2026.4.1 was ready on one card (0000:7c:00.0) and answered its warm-up. - B2's ladder: rung (a) stops on the missing setvars.sh, as the pre-registration read from the image. Rung (b) gets vLLM v0.28.0 started. Its four XPU workers then fail in their first all_reduce: oneCCL's init_device_fds cannot open /dev/dri/by-path, which neither the k3d node container nor the pod has. Rungs (c) and (d) do not apply to that error, so B2 is not deliverable. By the pre-registered rule no hold runs and H53 is INCONCLUSIVE. 0 of 8 holds were used. Serving proof after the restore: gpu-serving.sh answered on each pair's port, each pair's docker inspect key (image, command, env, devices, mounts, ports) matched its pre-stop copy, and a chat through Dynamo :8100 answered "OK" with each pair's num_requests_total rising (:30000 2 to 3, :30001 3 to 4). The technical report's counts move to 29 INCONCLUSIVE and 18 PENDING. --- EXPERIMENTS.md | 110 +++- .../results/2026-10-20/H53/prehold/fetch.log | 69 +++ .../2026-10-20/H53/prehold/frontdoor-post.txt | 2 + .../2026-10-20/H53/prehold/frontdoor-pre.txt | 2 + .../inspect-qwen38-a-hr.after.key.json | 1 + .../prehold/inspect-qwen38-a-hr.before.json | 491 +++++++++++++++++ .../inspect-qwen38-a-hr.before.key.json | 1 + .../inspect-qwen38-b-hr.after.key.json | 1 + .../prehold/inspect-qwen38-b-hr.before.json | 493 ++++++++++++++++++ .../inspect-qwen38-b-hr.before.key.json | 1 + .../results/2026-10-20/H53/prehold/setup.txt | 16 + .../results/2026-10-20/H53/prehold/verify.txt | 64 +++ .../H53/runs/H53-smoke-AG/dmesg-post.txt | 200 +++++++ .../H53/runs/H53-smoke-AG/dmesg-pre.txt | 200 +++++++ .../2026-10-20/H53/runs/H53-smoke-AG/hold.log | 41 ++ .../H53/runs/H53-smoke-AG/orphans_killed | 1 + .../2026-10-20/H53/runs/H53-smoke-AG/post.txt | 15 + .../2026-10-20/H53/runs/H53-smoke-AG/pre.txt | 15 + .../2026-10-20/H53/runs/H53-smoke-AG/rc | 1 + .../H53/runs/H53-smoke-AG/serving.jsonl | 29 ++ .../H53/runs/H53-smoke-AG/stale_reaped | 1 + .../2026-10-20/H53/runs/H53-smoke-AG/unreaped | 0 .../2026-10-20/H53/runs/H53-smoke-AG/wait_s | 1 + .../H53/runs/H53-smoke-B2-a/dmesg-post.txt | 200 +++++++ .../H53/runs/H53-smoke-B2-a/dmesg-pre.txt | 200 +++++++ .../H53/runs/H53-smoke-B2-a/hold.log | 7 + .../H53/runs/H53-smoke-B2-a/orphans_killed | 1 + .../H53/runs/H53-smoke-B2-a/post.txt | 15 + .../H53/runs/H53-smoke-B2-a/pre.txt | 15 + .../2026-10-20/H53/runs/H53-smoke-B2-a/rc | 1 + .../H53/runs/H53-smoke-B2-a/serving.jsonl | 31 ++ .../H53/runs/H53-smoke-B2-a/stale_reaped | 1 + .../H53/runs/H53-smoke-B2-a/unreaped | 0 .../2026-10-20/H53/runs/H53-smoke-B2-a/wait_s | 1 + .../H53/runs/H53-smoke-B2-b/dmesg-post.txt | 200 +++++++ .../H53/runs/H53-smoke-B2-b/dmesg-pre.txt | 200 +++++++ .../H53/runs/H53-smoke-B2-b/hold.log | 7 + .../H53/runs/H53-smoke-B2-b/orphans_killed | 1 + .../H53/runs/H53-smoke-B2-b/post.txt | 15 + .../H53/runs/H53-smoke-B2-b/pre.txt | 15 + .../2026-10-20/H53/runs/H53-smoke-B2-b/rc | 1 + .../H53/runs/H53-smoke-B2-b/serving.jsonl | 65 +++ .../H53/runs/H53-smoke-B2-b/stale_reaped | 1 + .../H53/runs/H53-smoke-B2-b/unreaped | 0 .../2026-10-20/H53/runs/H53-smoke-B2-b/wait_s | 1 + .../H53/smoke/H53-smoke-AG/hold/agent.err | 2 + .../H53/smoke/H53-smoke-AG/hold/agent.log | 48 ++ .../H53/smoke/H53-smoke-AG/hold/agent.sh | 13 + .../H53/smoke/H53-smoke-AG/hold/agent.yaml | 112 ++++ .../H53/smoke/H53-smoke-AG/hold/b-listing.txt | 34 ++ .../H53-smoke-AG/hold/blueprint-commit.txt | 1 + .../H53/smoke/H53-smoke-AG/hold/commit.txt | 1 + .../H53/smoke/H53-smoke-AG/hold/deploy.yaml | 98 ++++ .../H53/smoke/H53-smoke-AG/hold/gpu-arms.txt | 1 + .../H53/smoke/H53-smoke-AG/hold/gpu-hold.txt | 4 + .../H53/smoke/H53-smoke-AG/hold/image.txt | 1 + .../smoke/H53-smoke-AG/hold/litellm-image.txt | 1 + .../H53/smoke/H53-smoke-AG/hold/litellm.yaml | 7 + .../H53/smoke/H53-smoke-AG/hold/live.log | 13 + .../smoke/H53-smoke-AG/hold/load-start.log | 1 + .../H53/smoke/H53-smoke-AG/hold/mcp.log | 30 ++ .../H53/smoke/H53-smoke-AG/hold/models.json | 1 + .../H53/smoke/H53-smoke-AG/hold/pf.log | 5 + .../H53/smoke/H53-smoke-AG/hold/reset-mcp.sh | 7 + .../H53/smoke/H53-smoke-AG/hold/up.log | 33 ++ .../H53-smoke-AG/pairs/baseline-hold1.jsonl | 2 + .../smoke/H53-smoke-AG/pairs/facts-hold1.json | 349 +++++++++++++ .../H53-smoke-AG/pairs/proxy-AG-hold1.jsonl | 2 + .../H53-smoke-AG/pairs/registration.json | 1 + .../smoke/H53-smoke-AG/pairs/warmups.jsonl | 1 + .../smoke/H53-smoke-B2-a/hold/b-listing.txt | 34 ++ .../H53-smoke-B2-a/hold/blueprint-commit.txt | 1 + .../H53/smoke/H53-smoke-B2-a/hold/commit.txt | 1 + .../H53/smoke/H53-smoke-B2-a/hold/deploy.yaml | 189 +++++++ ...e-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt | 119 +++++ .../H53/smoke/H53-smoke-B2-a/hold/eat.yaml | 301 +++++++++++ .../H53/smoke/H53-smoke-B2-a/hold/events.txt | 163 ++++++ .../smoke/H53-smoke-B2-a/hold/gpu-hold.txt | 4 + .../H53/smoke/H53-smoke-B2-a/hold/live.log | 5 + ...en38-27b-xpu-799dc85d7c-ng77g.previous.txt | 1 + ...g-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt | 1 + .../H53/smoke/H53-smoke-B2-a/hold/pin.yaml | 2 + .../H53/smoke/H53-smoke-B2-a/hold/pods.txt | 10 + .../smoke/H53-smoke-B2-a/hold/reset-mcp.sh | 7 + .../H53/smoke/H53-smoke-B2-a/hold/up.log | 32 ++ .../smoke/H53-smoke-B2-b/hold/b-listing.txt | 34 ++ .../H53-smoke-B2-b/hold/blueprint-commit.txt | 1 + .../H53/smoke/H53-smoke-B2-b/hold/commit.txt | 1 + .../H53/smoke/H53-smoke-B2-b/hold/deploy.yaml | 188 +++++++ ...e-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt | 119 +++++ .../smoke/H53-smoke-B2-b/hold/devdri-diag.txt | 72 +++ .../H53/smoke/H53-smoke-B2-b/hold/eat.yaml | 301 +++++++++++ .../H53/smoke/H53-smoke-B2-b/hold/events.txt | 164 ++++++ .../smoke/H53-smoke-B2-b/hold/gpu-hold.txt | 4 + .../H53/smoke/H53-smoke-B2-b/hold/live.log | 5 + ...en38-27b-xpu-69c59cbbdc-5zbzz.attempt2.txt | 288 ++++++++++ ...en38-27b-xpu-69c59cbbdc-5zbzz.previous.txt | 288 ++++++++++ ...g-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt | 0 .../H53/smoke/H53-smoke-B2-b/hold/pin.yaml | 2 + .../H53/smoke/H53-smoke-B2-b/hold/pods.txt | 10 + .../smoke/H53-smoke-B2-b/hold/reset-mcp.sh | 7 + .../H53/smoke/H53-smoke-B2-b/hold/up.log | 33 ++ .../H53/window/H53-smoke-AG.launch.log | 11 + .../H53/window/H53-smoke-B2-a.launch.log | 11 + .../H53/window/H53-smoke-B2-b.launch.log | 11 + .../2026-10-20/H53/window/campaign-H53.log | 12 + .../results/2026-10-20/H53/window/deadman.log | 0 .../results/2026-10-20/H53/window/events.log | 23 + .../2026-10-20/H53/window/holds-H53.log | 6 + .../results/2026-10-20/H53/window/restore.log | 4 + .../2026-10-20/H53/window/serving-events.log | 8 + .../2026-10-20/H53/window/start.launch.log | 7 + .../results/2026-10-20/H53/window/stop.log | 4 + .../2026-10-20/H53/window/xe-prom.json | 1 + docs/intel/technical-report.md | 7 +- 115 files changed, 5989 insertions(+), 4 deletions(-) create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/fetch.log create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-post.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-pre.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.after.key.json create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.json create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.key.json create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.after.key.json create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.json create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.key.json create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/setup.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/prehold/verify.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-post.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-pre.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/hold.log create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/orphans_killed create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/post.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/pre.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/rc create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/serving.jsonl create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/stale_reaped create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/unreaped create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/wait_s create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-post.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-pre.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/hold.log create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/orphans_killed create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/post.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/pre.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/rc create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/serving.jsonl create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/stale_reaped create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/unreaped create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/wait_s create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-post.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-pre.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/hold.log create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/orphans_killed create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/post.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/pre.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/rc create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/serving.jsonl create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/stale_reaped create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/unreaped create mode 100644 bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/wait_s create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.err create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.log create mode 100755 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.sh create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.yaml create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/b-listing.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/blueprint-commit.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/commit.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/deploy.yaml create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-arms.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-hold.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/image.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm-image.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm.yaml create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/live.log create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/load-start.log create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/mcp.log create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/models.json create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/pf.log create mode 100755 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/reset-mcp.sh create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/up.log create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/baseline-hold1.jsonl create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/facts-hold1.json create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/proxy-AG-hold1.jsonl create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/registration.json create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/warmups.jsonl create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/b-listing.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/blueprint-commit.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/commit.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/deploy.yaml create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/describe-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/eat.yaml create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/events.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/gpu-hold.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/live.log create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.previous.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pin.yaml create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pods.txt create mode 100755 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/reset-mcp.sh create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/up.log create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/b-listing.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/blueprint-commit.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/commit.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/deploy.yaml create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/describe-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/devdri-diag.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/eat.yaml create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/events.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/gpu-hold.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/live.log create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.attempt2.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.previous.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pin.yaml create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pods.txt create mode 100755 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/reset-mcp.sh create mode 100644 bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/up.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/H53-smoke-AG.launch.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-a.launch.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-b.launch.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/campaign-H53.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/deadman.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/events.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/holds-H53.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/restore.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/serving-events.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/start.launch.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/stop.log create mode 100644 bench/eatflow/results/2026-10-20/H53/window/xe-prom.json diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 799802c1..5c42a3aa 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -70,7 +70,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H51 | (Model, arms and sampling superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20): Gemma 4 26B-A4B QAT.) Graded agents per box on the Xeon 6741P: H45's ramp, gates and scoring (B15-RO, typical load, 300 s steps, pooled p95 turn <= 10 s, success >= the arm's N=1 success - 10 points, errors <= 1%, >= 60 window requests), extended to N=16, for A1 and B1. | Superseded by [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20), which restates the predictions; as first written: Written 2026-10-20, before any measurement. A1 in [1, 8] (0.5). B1 in [0, 4] (0.5). B1 <= A1 (0.55). | H45's `apb` runner and scorer (`apb-score --hyp H51`, ramp 1, 2, 4, 8, 16), one fresh cluster per hold, at most 4 holds of at most 40 min. A prediction holds when the arm's bracket lies inside its range and is refuted when it lies outside; B1 <= A1 is decided only on matched load. REFUTED when any prediction is refuted, SUPPORTED when all three hold, else INCONCLUSIVE. Runs only if H50's holds end within their cap. Model and predictions from [amendment M1](#crescent-amendment-m1-gemma-4-26b-a4b-qat-on-the-cpu-rows-2026-10-20). B1 as H50, from [amendment E1](#crescent-amendment-e1-eat-stock-is-release-110-for-h50-h50o-and-h51-2026-10-29); node image, deploy commit and order per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); deploy commit, M1's ranges in the scorer, one hold per arm and the wiring smokes per [H51 amendment S1](#h51-amendment-s1-deploy-commit-scorer-ranges-ramp-per-hold-and-wiring-smokes-2026-10-29). Result ([H51 run](#h51-run-xeon-6741p-2026-10-29)): one hold per arm, 2 of the 4 permitted, none VOID. Both arms failed N=1. A1: 65 window requests, turn p95 10.41 s against the 10 s SLO, success 30/32. B1: 58 window requests (under 60), p95 11.52 s, success 27/29. Agents per box A1 0, outside its predicted [1, 4], so REFUTED. B1's 0 lies inside [0, 4], and B1 <= A1 holds on matched load (0 <= 0). | REFUTED | | H52 | A plain `k3d-gpu up` (no flags, the auto-adapt default) on the Xeon 6741P host as found detects and enables every engine actually present and advertises each usable one to the scheduler, with nothing missing and nothing phantom; engines present but not configured on the host are named, not advertised. | Written 2026-10-20 from an inventory read from sysfs, lspci and CPUID, not from k3d-gpu. 4 x Arc Pro B70 (xe) as `gpu.intel.com/xe: 4`; AMX TILE, BF16, INT8, FP16 and 14 AVX-512/VNNI labels; no hybrid, NUMA, NVIDIA, Gaudi or NPU objects; DSA, IAA and QAT named as present but unconfigured (0.4: `up` may not name them). | The table in [H50-H52 pre-registration](#h50-h52-pre-registration-the-xeon-6741p-2026-10-20): `k3d-gpu doctor`, `K3D_GPU_CLUSTER=k3dgpu-h52 k3d-gpu up`, `kubectl get node -o json`, `k3d-gpu test`, and one enumeration pod per B70. SUPPORTED: every row matches. REFUTED: any row does not. Run 2026-10-20 ([H52 run](#h52-run-xeon-6741p-2026-10-20)): the four B70s (`gpu.intel.com/xe: 4`, `xmx=true`), AMX (TILE, BF16, INT8, FP16) and all 14 AVX-512/VNNI labels detected and reached pods, nothing phantom; but no media engine label on `xe`, and `up` does not name the unconfigured DSA, IAA and QAT (`doctor` does). | REFUTED | | H52b | After `k3d-gpu host-setup --apply` configures the host's DSA, IAA and QAT (the fix that adds that path, merged first), a plain `k3d-gpu up` also advertises them, and each one reaches a pod; the host is then restored exactly. | Written 2026-10-20, before the fix exists. DSA, IAA and QAT allocatable > 0, each at the count host-setup reports configuring; their test pods see `/dev/dsa`, `/dev/iax`, `/dev/vfio` (0.6). The other rows read as H52's expected table, media labels included ([amendment D1](#h52-amendment-d1-the-fixes-and-h52b-2026-10-20)). | Runs after H52, once the host-setup fix is merged; same procedure plus `host-setup --apply` before `up` and its printed revert after. SUPPORTED: the three engines advertised and reached, the H52 rows unchanged, and the host inventory read back equal to the one taken before. REFUTED: any of these fails. The host-setup command (`--apply --with-kernel-wqs`), the plan read first, the counts, the plain `up`'s node image and the revert per [amendment D2](#h52-amendment-d2-h52bs-host-setup-step-on-crescent-2026-10-29); the runner and deploy commit per [amendment D3](#h52-amendment-d3-h52bs-runner-and-deploy-commit-2026-10-29). Result ([H52b run](#h52b-run-xeon-6741p-2026-10-29)): one hold at k3d-gpu 0.1.79; the plan passed D2's gate (8 commands) and applied. DSA advertised 3 of the 3 configured work queues and its pod saw `/dev/dsa`; IAA got no work queue and no resource (as D2 item 4 disclosed); QAT got 48 VFs, but its plugin crash-looped and advertised none; `gpu.intel.com/xe` read 2, not 4: the serving pairs hold about half the VRAM of all four cards, and `up`'s busy-card filter (half the VRAM) left out the two just over half (16.1, 16.2 GiB) and kept the two just under it (15.4, 15.5 GiB), which the pairs also hold, so two enumeration pods ran, not four. Media labels (`vcs` 2, `vecs` 2), AMX, the 14 AVX-512/VNNI labels, CPU 96 and no other vendor read as expected. The third inventory equals the first. Three clauses fail, so REFUTED. | REFUTED | -| H53 | Inside EAT's banking flow on the Arc Pro B70s with Qwen3.8-27B (thinking off), k3d-gpu's own GPU choice AG (OVMS int4 on one card, `agent manifest`, auto device) delivers far more B24 tasks within the 120 s turn budget than B2, "EAT PR #21 nearest profile (Qwen3.6-27B) on Qwen3.8 weights, fp16" (vLLM XPU v0.28.0, TP 4). A stack-vs-stack claim. | Written 2026-10-20, before any H53 measurement. AG >= 0.70 (0.6); B2 boots by ladder rung (b) (0.6); B2 >= 0.50 (0.5); block mean AG - B2 in [-0.15, +0.30] (0.5). INCONCLUSIVE (0.6), REFUTED (0.3), SUPPORTED (0.1). | Solo holds in the order AG, B2, B2, AG, AG, B2 (up to 8), blocks = hold pairs (2k-1, 2k), block-level inference: mean of per-block d over >= 3 counted blocks, t-interval. SUPPORTED: lower bound >= +30 points and AG >= 0.6, L3 gates passed. REFUTED: upper bound < +30 points. Else INCONCLUSIVE. `bench/eatflow/crescent/solo-hold.sh N ARM`, `agentbench solo-report --hyp H53 --arm-a AG --arm-b B2`. See [H53 pre-registration](#h53-pre-registration-eats-banking-flow-on-arc-pro-b70-2026-10-20). Node image, deploy commit, order and the restore check per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); the solo runner fix, setup, the wiring smokes, the window and its futility stop per [H53 amendment R1](#h53-amendment-r1-the-solo-runner-fix-setup-wiring-smokes-and-the-gpu-window-2026-10-30); tools per [amendment E4](#crescent-amendment-e4-kubectl-v1372-helm-v431-and-uv-0136-in-the-crescent-workspace-2026-10-29). | PENDING | +| H53 | Inside EAT's banking flow on the Arc Pro B70s with Qwen3.8-27B (thinking off), k3d-gpu's own GPU choice AG (OVMS int4 on one card, `agent manifest`, auto device) delivers far more B24 tasks within the 120 s turn budget than B2, "EAT PR #21 nearest profile (Qwen3.6-27B) on Qwen3.8 weights, fp16" (vLLM XPU v0.28.0, TP 4). A stack-vs-stack claim. | Written 2026-10-20, before any H53 measurement. AG >= 0.70 (0.6); B2 boots by ladder rung (b) (0.6); B2 >= 0.50 (0.5); block mean AG - B2 in [-0.15, +0.30] (0.5). INCONCLUSIVE (0.6), REFUTED (0.3), SUPPORTED (0.1). | Solo holds in the order AG, B2, B2, AG, AG, B2 (up to 8), blocks = hold pairs (2k-1, 2k), block-level inference: mean of per-block d over >= 3 counted blocks, t-interval. SUPPORTED: lower bound >= +30 points and AG >= 0.6, L3 gates passed. REFUTED: upper bound < +30 points. Else INCONCLUSIVE. `bench/eatflow/crescent/solo-hold.sh N ARM`, `agentbench solo-report --hyp H53 --arm-a AG --arm-b B2`. See [H53 pre-registration](#h53-pre-registration-eats-banking-flow-on-arc-pro-b70-2026-10-20). Node image, deploy commit, order and the restore check per [amendment E2](#crescent-amendment-e2-pins-deploy-commits-order-and-the-restore-check-for-the-pending-crescent-rows-2026-10-29); the solo runner fix, setup, the wiring smokes, the window and its futility stop per [H53 amendment R1](#h53-amendment-r1-the-solo-runner-fix-setup-wiring-smokes-and-the-gpu-window-2026-10-30); tools per [amendment E4](#crescent-amendment-e4-kubectl-v1372-helm-v431-and-uv-0136-in-the-crescent-workspace-2026-10-29). Result ([H53 run](#h53-run-arc-pro-b70-2026-10-30)): no hold ran, 0 of the 8 permitted. AG's wiring smoke passed (OVMS on one card, warm-up answered). B2 needed rung (b) (`setvars.sh` is not in the v0.28.0 image), then failed in its first cross-card collective: oneCCL opens `/dev/dri/by-path`, which the k3d node and pod lack. No further rung applies, so B2 is not deliverable and H53 is INCONCLUSIVE by its pre-registered rule. | INCONCLUSIVE | | H54a | On the Xeon 6741P, Gemma 4 26B-A4B (QAT weights) single-stream: the best optimized CPU arm, chosen by the pre-registered source and speed rule, decodes at N=1 at least 3x faster than EAT stock B1g (vLLM CPU v0.24.0 bf16, `xeon-values.yaml`'s Gemma entry on the QAT bf16 weights), with a 50-item quality spot-check no more than 10 points below B1g's. A microbench, not an agent claim. | Written 2026-10-20, before any H54 measurement. N=1 median decode tok/s: B1g in [4, 15] (0.6); Lg (llama.cpp CPU, Google QAT Q4_0) in [25, 70] (0.6); A1g (k3d-gpu auto, OVMS CPU, QAT-matched IR) in [15, 60] (0.5), A1g/A1o in [0.85, 1.02] (0.6); Vq (vLLM CPU v0.32.0 bf16) in [6, 18] (0.5), Vq-mtp/Vq in [1.4, 2.5] (0.5); Lg-mtp/Lg in [1.2, 2.2] (0.4); Sg (SGLang CPU AMX) deliverable (0.6). The chosen arm is Lg or Lg-mtp (0.55). B3g not deliverable (0.8). SUPPORTED (0.65), INCONCLUSIVE (0.25), REFUTED (0.1). Not gated: TTFT on a 1,000-token prompt, aggregate tok/s at N=4 and 16 (A1g N=16/N=1 <= 1.5, 0.5), J/token. | [H54 pre-registration](#h54-pre-registration-gemma-4-26b-a4b-qat-and-qwen38-microbenches-on-crescent-2026-10-20): `bench/llm_load.py` per arm (N=1: 8 requests, N=4: 16, N=16: 32, 256 tokens, greedy, `ignore_eos`), `bench/energy.sh` RAPL per phase, GSM8K items 0-49 graded by exact match, one arm serving at a time, at most 3 holds of at most 40 min. Statistic: ratio of the chosen arm's median N=1 decode to B1g's, 95% bootstrap CI over requests. SUPPORTED: CI lower bound >= 3.0 and the quality gate passed. REFUTED: CI upper bound < 3.0, or every arm with a point ratio >= 3.0 fails the quality gate. Otherwise INCONCLUSIVE (also: B1g not deliverable, or a void phase not re-run). Run 2026-10-21, 3 holds ([H54a run](#h54a-run-xeon-6741p-2026-10-21)): stock B1g stalled in both its holds (vLLM v0.24.0 `EngineDeadError` mid-decode, cause open; no scored phase, 0/50 GSM8K, all errored), so `h54.py score-a` has no base and no chosen arm. Measured: Lg 48.74, Lg-mtp 54.52, A1g 23.25, A1o 21.73 tok/s N=1 median, GSM8K 50, 50, 49, 49 of 50; each scored phase exists only under amendment R4 (its own foreign CPU 4.4-4.8 CPU-s/s). Vq, Vq-mtp, Vi, Sg, B3g not run (cap). Descriptive, not scored: Lg-mtp over B1g's 2 hold-2 rates 3.02 ~~(CI 2.90-3.06)~~ (descriptive only: n=2 base rates from an engine that died minutes later, no CI; correction 2026-10-28, [H54a run](#h54a-run-xeon-6741p-2026-10-21)). | INCONCLUSIVE | | H54q | QAT matters at int4 on this model: on GSM8K items 0-199, the QAT-matched OpenVINO IR OVQ (INT4_SYM, group 32, built from Google's QAT bf16 weights) is non-inferior to the bf16 QAT reference R within 5 points, and the published base-model IR OVB (`OpenVINO/gemma-4-26b-a4b-it-int4-ov`; ~~INT4_ASYM group 128~~ AWQ INT4 asymmetric group 64, router layers unquantized, per its `openvino_config.json`: [note N1](#h54-note-n1-what-ovb-is-2026-10-21)) scores below OVQ. | Written 2026-10-20, before any measurement. R accuracy in [0.85, 0.96] (0.6). OVQ - R in [-0.03, +0.02] (0.6). GGUF Q4_0 (Lg) - R in [-0.03, +0.02] (0.6). OVB - R in [-0.07, 0] (0.5). The OVB < OVQ part resolves on 200 items (0.2). INCONCLUSIVE (0.65), SUPPORTED (0.2), REFUTED (0.15). | As H54a's quality protocol on items 0-199 of `openai/gsm8k` `main` test @ `740312ad`; items 0-49 come from H54a's runs of the same arm configs; ~~at most 1 hold of at most 40 min for the rest~~ the rest in at most 3 holds of at most 40 min, R answering all 200 items because H54a never ran its config ([amendment Q1](#h54-amendment-q1-h54qs-reference-arm-hold-cap-and-order-2026-10-29)). Paired bootstrap 95% CI over items (10,000 resamples, seed 1). SUPPORTED: CI lower bound of OVQ - R >= -0.05 and CI upper bound of OVB - OVQ < 0. REFUTED: CI upper bound of OVQ - R < -0.05, or CI lower bound of OVB - OVQ > 0. Otherwise INCONCLUSIVE. | PENDING | | H54b | On Arc Pro B70 with Qwen3.8-27B single-stream: the best optimized arm on ONE card (k3d-gpu auto OVMS int4, Intel's llm-scaler vLLM, or llama.cpp SYCL, with and without each one's documented MTP) decodes at N=1 at least 2x faster than EAT #21's XPU path B2q on TWO cards (vLLM XPU v0.28.0, the `xpu-qwen36-27b` profile's flags on the Qwen3.8-27B weights, TP 2). | Written 2026-10-20, before any measurement. N=1 median decode tok/s: X1q (OVMS GPU int4) in [15, 28] (0.5), X1q-mtp/X1q in [1.3, 2.2] (0.4); Sq (llm-scaler 1.5, online FP8, TP 1) deliverable (0.5), in [10, 18]; Sq-mtp/Sq in [1.4, 2.3] (0.5); Lsq (llama.cpp SYCL, Intel Q4_K_M) in [12, 24] (0.5); B2q deliverable at TP 2 (0.4), in [5, 12] (0.5). Chosen/B2q >= 2 (0.6). Cards free for a window (0.5). INCONCLUSIVE (0.6), SUPPORTED (0.3), REFUTED (0.1). | H54a's protocol on the B70s, cards only when `~/k3dgpu-bench/gpu-state.txt` lists them free; at most 2 holds of at most 40 min. Same statistic and quality gate (against B2q, or the official-weights arm Sq when B2q is not deliverable, for the gate only). SUPPORTED: CI lower bound of chosen one-card arm / B2q >= 2.0 and the gate passed. REFUTED: CI upper bound < 2.0. Otherwise INCONCLUSIVE (also: B2q not deliverable, or no free card before the campaign ends). Gemma 4 QAT on one B70 (X1g, X1o, B2g, Sgg) is described if hold time remains, never gated. Run 2026-10-21, 2 holds ([H54b and H54c run](#h54b-and-h54c-run-arc-pro-b70-2026-10-21)): INCONCLUSIVE, not delivered as pre-registered: the pinned "one card" pods saw all 4 cards. B2q, Sq, Sq-mtp (Qwen bf16, 80.6 GB with its guard) and Rq were not deliverable (disk); X1q and X1q-mtp failed on two `h54b.sh` runner bugs before any engine started; Lsq ran, but its pod listed every render node and llama.cpp split the model over all four B70s, two of them serving the live `qwen38-a-hr`, so no one-card arm exists. Lsq-mtp was not run: its GGUF carries the MTP (`nextn`) tensors, but G1's runner has no such arm; nor were the described-only Gemma arms. `score.py`: "no gate reference (B2q and Sq missing)". Descriptive only, a 4-card run and not a one-card or B70 result: Lsq 27.3 tok/s N=1 median, GSM8K 48/50. | INCONCLUSIVE | @@ -4435,6 +4435,114 @@ on a B70 or serves Qwen3.8-27B. `smoke//`. The score is `agentbench solo-report bench/eatflow/results/2026-10-20/H53/pairs --hyp H53 --arm-a AG --arm-b B2`, at the deploy commit. +### H53 run (Arc Pro B70, 2026-10-30) + +**Reproduce:** + +```text +# provenance +date: 2026-10-30T00:13:39Z to 00:38:03Z; GPU window 00:14:50Z (both pairs stopped by 00:15:06Z) + to 00:38:03Z (both answering at 00:37:43Z); no scored hold ran +hypothesis: H53 +host cpu: Intel(R) Xeon(R) 6741P (Granite Rapids), 48 cores / 96 threads, 503 GiB RAM +host kernel: 7.0.0-31-generic; Docker 29.8.0 +accelerators: 4x Intel Arc Pro B70 (PCI 8086:e223, driver xe) at 0000:12:00.0, 0000:48:00.0, + 0000:7c:00.0 and 0000:96:00.0, all four free for the window; k3d-gpu up: intel=4 amx=1 +k3d-gpu commit: 1f39488b (branch claude/h53-run: origin/main 74ff63ef, k3d-gpu 0.1.80, plus amendment R1), + checked out clean and detached at ~/k3dgpu-bench/k3d-gpu-h53 +node image: cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c; + k3d v5.9.0, kubectl v1.37.2, helm v4.3.1, uv 0.13.6 (amendment E4's setup.sh, run at 1f39488b + under the bench lock, every checksum OK: prehold/setup.txt) +model: AG OpenVINO/Qwen3.8-27B-int4-ov @ 7b20ea8c; B2 Qwen/Qwen3.8-27B @ 1d4bf0f2 (bf16, mounted + read-only, listing equal to setup's); every LFS file of both equal to the Hub's sha256 at + download (prehold/fetch.log) and again before the window (prehold/verify.txt) +runtime: AG `agent manifest` (auto device, default draft rule): openvino/model_server@sha256:a928903f... + (2026.4.1), --target_device=GPU, gpu.intel.com/xe 1, tool_parser qwen3coder, no reasoning + parser, no draft; B2 EAT release-1.1.0 XPU path (the c86c42b tree) core/helm-charts/vllm + xpu-values.yaml, LLM_MODEL_ID=Qwen/Qwen3.6-27B, vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a..., + tensor_parallel_size 4, gpu.intel.com/xe 4; LiteLLM v1.83.14-stable.patch.2@sha256:73cbb215...; + blueprint 758d484, ENABLE_MEMORY=false; sampling agent_qwen38 (temperature 0.7, top_p 0.8, + top_k 20, presence_penalty 1.5, max_tokens 1024, thinking off) +load: typical, in the AG smoke only (the B2 smokes ended before load starts) +runs per arm: 0 of the 8 permitted holds; 3 wiring smokes, discarded +``` + +Commands, on crescent, from the H53 checkout: +`bench/eatflow/results/2026-10-20/H53/prehold/prehold.sh` steps `setup`, `verify`, `frontdoor pre`, +`stop`, `smoke AG`, `smoke B2`, `smoke B2 b`, then `start`. No `holds` step ran, so there is no +`solo-report` score. Records: [bench/eatflow/results/2026-10-20/H53/](bench/eatflow/results/2026-10-20/H53/) +(`prehold/`, `window/` with `events.log` and `xe-prom.json`, `smoke/` with each smoke's records and pod +logs, `runs/` with each smoke's `hold.sh` records). + +**Before the window, all at `1f39488b`:** setup.sh installed E4's four tools, every checksum OK +(00:13:39-00:13:42Z). The re-verify read every LFS file of both models equal to the Hub's sha256, and B2's +listing equal to setup's (rc 0 for each). Through Dynamo `:8100`, each serving pair answered "OK" and its +`num_requests_total` rose by one: :30000 161 to 162, :30001 57 to 58 (`prehold/frontdoor-pre.txt`, 00:14:45Z). + +**The window.** `stop` saved `docker inspect` of both pairs, then stopped `qwen38-a-hr` (00:15:00Z) and +`qwen38-b-hr` (00:15:06Z) with `docker stop`. The cards read 16.14, 15.43, 16.18 and 15.47 GiB used before +the stop and 0.0-0.17 GiB after it. + +- **AG smoke** (`H53-smoke-AG`, 00:15:13-00:20:00Z): passed. `agent manifest` chose the GPU + (`--target_device=GPU`, `gpu.intel.com/xe: 1`; its fit: one GPU, context 32768 x 6 sequences). OVMS loaded the + IR as a VLM continuous-batching servable and was ready 51 s after its deploy (241 s into the smoke, after + `up` and the 16 GB IR copy). Its pod held one card, 0000:7c:00.0 (`gpu-arms.txt`). The warm-up answered in + 5.5 s. Prometheus read 14.81 GiB peak on that card and 0.00 on the other three. +- **B2 rung (a)** (`H53-smoke-B2-a`, 00:20:14-00:25:26Z): failed. The container exits at once with + `/opt/intel/oneapi/setvars.sh: No such file or directory`, which is rung (b)'s error, as the + pre-registration had read from the image (item 2). Stopped early after two restarts. +- **B2 rung (b)** (`H53-smoke-B2-b`, 00:25:35-00:36:28Z): failed. Without the `setvars.sh` source vLLM + v0.28.0 starts, and its four XPU workers then fail in their first collective + (`xpu_worker.init_device`, `torch.distributed.all_reduce`): + `RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open + device directory`. The engine core then exits with "Engine core initialization failed". It is the same + error at 00:33:38Z and 00:35:27Z (two saved attempt logs), and the pod restarted three times. The pod held + all four cards and touched each (0.51-0.84 GiB peak) before the failure. +- **Why rung (b) fails** (`smoke/H53-smoke-B2-b/hold/devdri-diag.txt`, read while the pod ran): the image's + oneCCL (`/opt/venv/lib/libccl.so.1`) opens `/dev/dri/by-path/`. The host has that directory. The k3d + node container does not: it gets `/dev/dri/cardN` and `renderDN` as devices, which carry no `by-path` + symlinks. The Intel GPU plugin mounts the node's `/dev/dri`, so the pod's `/dev/dri` has the four card + and render nodes and no `by-path`. The gap is in k3d-gpu's device exposure, not in EAT. This run shows it + for B2's oneCCL settings (`CCL_ZE_IPC_EXCHANGE=pidfd`, from the chart); another multi-card oneCCL job that + opens the same directory would meet it too. A one-card arm makes no collective, which is why AG is + unaffected. +- **The ladder ends at (b).** Rung (c) drops `--disable-log-requests`, which B2's command does not carry. + Rung (d) applies only to a KV-cache failure. Neither applies to this error, and the pre-registration + allows no other change. So B2 does not answer, and the pre-registration's rule applies: "If B2 does not + answer after (d), H53 is INCONCLUSIVE (not deliverable) and the failure is the finding" (R1 item 4: no + hold runs). + +`start` restarted both pairs at 00:36:38Z. `gpu-serving.sh` read `/health` 200, `/v1/models` and a 16-token +chat from `qwen38-b-hr` at 00:37:40Z and `qwen38-a-hr` at 00:37:43Z. Each pair's image, command, env, +devices, mounts and ports equal its inspect from before the stop. Through `:8100`, each pair answered "OK" +and its `num_requests_total` rose: :30000 2 to 3 (00:37:52Z), :30001 3 to 4 (00:38:03Z) +(`prehold/frontdoor-post.txt`). The serving outage was 22.7 min (`qwen38-a-hr`, 00:15:00-00:37:43Z) and 22.6 min +(`qwen38-b-hr`, 00:15:06-00:37:40Z). No `xe` kernel message was logged during the window (`runs/*/dmesg-*.txt`). Every smoke's +cleanup deleted its cluster, the deadman exited with the window, and the bench lock was free afterwards. + +**Predictions, read as written:** + +- AG >= 0.70 (0.6): not measured. No hold ran. +- B2 boots after at most rung (b) (0.6) and answers (0.75): **missed**. B2 did not boot after rung (b), + and it did not answer. +- B2 >= 0.50 (0.5), block mean AG - B2 in [-0.15, +0.30] (0.5), three blocks count (0.7): not measured. +- Verdict INCONCLUSIVE (0.6): matched, but by not-deliverable rather than by a wide three-block interval. + +**Verdict: INCONCLUSIVE (not deliverable).** EAT release-1.1.0's XPU path for a 27B Qwen at TP 4 did not +start in a k3d-gpu cluster on this host. It needed rung (b) (its command sources a `setvars.sh` the v0.28.0 +image lacks), and then it failed on k3d-gpu's device exposure: the node has no `/dev/dri/by-path`, which +oneCCL needs for collectives across cards. k3d-gpu's one-card choice, AG, deployed and answered. No block +ran, so H53 says nothing about the stack-vs-stack claim. + +Scope: one host, one node image (0.1.61) and one k3d-gpu commit (`1f39488b`). The by-path gap is k3d-gpu's, +not EAT's. On a host where pods see `/dev/dri/by-path`, B2 might pass rung (b), which H53 does not test. + +Disclosed: amendment R1 was pushed at 00:13:25Z, before the first H53 step (`setup`, 00:13:39Z), and was not +merged before the run. Its futility rule and per-hold placement records did not bind, since no hold ran. +The model download (R1 item 8) ran on 2026-10-29 before R1 existed and measures nothing. The per-card +VRAM and engine record of the window (`window/xe-prom.json`) was read from Prometheus after the window, +read-only. + ### H54 pre-registration: Gemma 4 26B-A4B QAT and Qwen3.8 microbenches on crescent (2026-10-20) Written 2026-10-20, before any H54a, H54q, H54b, H54c or H54d measurement. Nothing here is a diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/fetch.log b/bench/eatflow/results/2026-10-20/H53/prehold/fetch.log new file mode 100644 index 00000000..b50e8c86 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/fetch.log @@ -0,0 +1,69 @@ +lock-taken 2026-10-29T21:44:37Z +6d5cab69b902d2d4b83d95af218c1ea069d0d9fc +Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. +Error while downloading from https://huggingface.co/Qwen/Qwen3.8-27B/resolve/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/model-00002-of-00018.safetensors: peer closed connection without sending complete message body (received 1344965215 bytes, expected 3043080328) +Trying to resume download... +✓ Downloaded + path: /home/devcloud/k3dgpu-bench/models/Qwen3.8-27B +Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. +✓ Downloaded + path: /home/devcloud/k3dgpu-bench/models/Qwen3.8-27B-int4-ov +ok size 1570 .gitattributes +ok size 11544 LICENSE +ok size 65012 README.md +ok size 8952 chat_template.jinja +ok size 4312 config.json +ok size 238 crc32.txt +ok size 202 generation_config.json +ok size 3353259 merges.txt +ok ba0ce20aae489ad196733da5064bcdf159a1fe84f53336648196e1ebb7751b1c model-00001-of-00018.safetensors +ok 06a148c01bfbe3faa14a5f184a7ff29a706f7ae1c8b2705d2058e26d17a001fb model-00002-of-00018.safetensors +ok 2e1bf62cbcd406eaa64b60d10353e1f0ef4039d0976e56f05cabe953454f9968 model-00003-of-00018.safetensors +ok 511e34063187882659753c4d93f3859f93c019fd438d8813071921c81d9a3f1a model-00004-of-00018.safetensors +ok 635cb53446dc74f219740fc59e18b774f877b803b9722e289ca62575a6efa701 model-00005-of-00018.safetensors +ok 0bc5214fac607f0e6cc92eec3789d4b8559410ef9fce66621ba8158e8410dae0 model-00006-of-00018.safetensors +ok 80b0c49033e9a0d5762562aa12f4acdb7f54da586f3d0110f28c48d91cf07892 model-00007-of-00018.safetensors +ok 7192c5b66185d3592927daabee1cc19e6f6e0ce75988ee20e824b624765fda79 model-00008-of-00018.safetensors +ok af3c48cc37af44f3db6ae0579baf019180d48d9c527caa0a1f03ff85813a56d8 model-00009-of-00018.safetensors +ok 163490a76f3bea3a40855b7efc04ce6d27afaf1a34f0bbde495b9491f76457c9 model-00010-of-00018.safetensors +ok 5f3ae1b948aeee39da77aec558e8236cd65fe4d7cb7686a76bb007acc563c6d8 model-00011-of-00018.safetensors +ok a3de1c7114677a8f5ac5c4892c90e8238ea5c1e2038c80e757dfc87c3902ca55 model-00012-of-00018.safetensors +ok 06ab79a41f74c9c5cb734816feb0c7fc364104b227165ee7391231e1155aa02a model-00013-of-00018.safetensors +ok 4138ed94603065ba884bbcadedb04d7718bb40117e85e6f5c6fc5b9c05b7a85b model-00014-of-00018.safetensors +ok 69224e27b9de4e7dbf6fc936c6eaae08447bda3b80a6c31a871ab451173afd22 model-00015-of-00018.safetensors +ok 73cb9a1089fb6155cb648609478d6633be8a5c7d9ca5a05bc8925ce8a553cefe model-00016-of-00018.safetensors +ok beb51f01056142ac4984bd800507b0dd0fd18de57f8e9ef6ea41d1a3598983a8 model-00017-of-00018.safetensors +ok 1d3479509e21494658f9b64d317f5ea8e55c4025d28c702d6c4d0b356ce8ea06 model-00018-of-00018.safetensors +ok size 112216 model.safetensors.index.json +ok size 390 preprocessor_config.json +ok 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3 tokenizer.json +ok size 17928 tokenizer_config.json +ok size 385 video_preprocessor_config.json +ok size 6722759 vocab.json +ok size 1570 .gitattributes +ok size 8321 README.md +ok size 8952 chat_template.jinja +ok size 3679 config.json +ok size 213 generation_config.json +ok size 1790 openvino_config.json +ok 48fc2c6a7a1056994982911eb9695826acef933c3d742364c56d16317a0b2a79 openvino_detokenizer.bin +ok size 16771 openvino_detokenizer.xml +ok 95aac72ac7286707c5e0fd9b120ea7f22c93f3ae17e7f1263442587d1d887d50 openvino_language_model.bin +ok size 9736672 openvino_language_model.xml +ok bc8ef47f252ad5a72be11e48ba6a23da50a67d1cef03396a7d902ca9aa1d5bc8 openvino_mtp_model.bin +ok size 210992 openvino_mtp_model.xml +ok e2099d0d83dcdfb00c378b51b50a52d28116aee71eb4f81cef0fe19ae37a085f openvino_text_embeddings_model.bin +ok size 5920 openvino_text_embeddings_model.xml +ok ca1ff37f580b558f0119b57824240adc92624d6d2ef6dd96ca2a24f4f1368c05 openvino_tokenizer.bin +ok size 35358 openvino_tokenizer.xml +ok 3792b6946c1f475aaac5d6f3960c7b912f612104e4f39ad68643b707370ed8e7 openvino_vision_embeddings_merger_model.bin +ok size 1330718 openvino_vision_embeddings_merger_model.xml +ok 5a8661b052abd07563d0f7148934d1fb77d5a842647151304287beb5dcee5541 openvino_vision_embeddings_model.bin +ok size 8610 openvino_vision_embeddings_model.xml +ok 82ccc97be0351cd82ebe1717fd7fa9adf72a44982a7ced8e8a845d869325d9e5 openvino_vision_embeddings_pos_model.bin +ok size 5887 openvino_vision_embeddings_pos_model.xml +ok size 390 preprocessor_config.json +ok size 1300 processor_config.json +ok 87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4 tokenizer.json +ok size 1098 tokenizer_config.json +rc=0 2026-10-29T21:55:52Z diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-post.txt b/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-post.txt new file mode 100644 index 00000000..5e145d28 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-post.txt @@ -0,0 +1,2 @@ +2026-10-30T00:37:52Z :30000 pass=1 before=2 after=3 content=OK +2026-10-30T00:38:03Z :30001 pass=1 before=3 after=4 content=OK diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-pre.txt b/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-pre.txt new file mode 100644 index 00000000..16e515a9 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/frontdoor-pre.txt @@ -0,0 +1,2 @@ +2026-10-30T00:14:45Z :30000 pass=1 before=161 after=162 content=OK +2026-10-30T00:14:45Z :30001 pass=1 before=57 after=58 content=OK diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.after.key.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.after.key.json new file mode 100644 index 00000000..63bd34fe --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.after.key.json @@ -0,0 +1 @@ +{"args": ["/entry.sh"], "binds": ["/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro", "/dev/dri:/dev/dri", "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", "/home/devcloud/hicache-shared:/hicache", "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", "/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro"], "devices": [{"CgroupPermissions": "rwm", "PathInContainer": "/dev/dri", "PathOnHost": "/dev/dri"}], "env": ["CCL_CONFIGURATION=cpu_gpu_dpcpp", "CCL_CONFIGURATION_PATH=", "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", "CTX=8192", "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", "DEBIAN_FRONTEND=noninteractive", "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", "DPLROOT=/opt/intel/oneapi/dpl/2022.13", "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", "KV_PORT=5557", "LANG=C.UTF-8", "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", "MKLROOT=/opt/intel/oneapi/mkl/2026.1", "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", "ONEAPI_ROOT=/opt/intel/oneapi", "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", "PORT=30000", "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", "SETVARS_COMPLETED=1", "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", "SGLANG_CHUNKED_PREFILL=4096", "SGLANG_CONTEXT_LEN=262144", "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", "SGLANG_HTTP_PORT=30000", "SGLANG_MAX_RUNNING=64", "SGLANG_MEM_FRACTION=0.85", "SGLANG_MODEL=/model", "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", "SGLANG_PAGE_SIZE=64", "SGLANG_TP_SIZE=2", "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", "TP_SIZE=2", "UMF_ROOT=/opt/intel/oneapi/umf/1.1", "UV_PYTHON_INSTALL_DIR=/opt/uv/python", "VIRTUAL_ENV=/opt/venv", "XPU_DENSE_INT8=1", "XPU_DENSE_INT8_MAX_M=8", "XPU_GDN_DECODE_BV=16", "XPU_KERNELS=topk,gemv", "XPU_MAMBA_PAGED=1", "XPU_MOE_ALIGN_SKIP=1", "XPU_MOE_BUILD=/opt/xpu/build-gemv", "XPU_REPO=/opt/xpu", "XPU_SYMM_AR=1", "XPU_SYMM_AR_MAX_BYTES=131072", "XPU_TOPK_BUILD=/opt/xpu/build", "XPU_UNROLL_SPEC_VERIFY=1", "ZE_AFFINITY_MASK=0,1"], "image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", "network": "bridge", "path": "bash", "ports": {"30000/tcp": [{"HostIp": "", "HostPort": "30000"}], "5557/tcp": [{"HostIp": "", "HostPort": "5557"}]}, "restart": {"MaximumRetryCount": 0, "Name": "no"}} diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.json new file mode 100644 index 00000000..21e2ee78 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.json @@ -0,0 +1,491 @@ +[ + { + "Id": "e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127", + "Created": "2026-10-19T07:43:07.036551199Z", + "Path": "bash", + "Args": [ + "/entry.sh" + ], + "State": { + "Status": "running", + "Running": true, + "Paused": false, + "Restarting": false, + "OOMKilled": false, + "Dead": false, + "Pid": 3898329, + "ExitCode": 0, + "Error": "", + "StartedAt": "2026-10-19T07:43:07.057267648Z", + "FinishedAt": "0001-01-01T00:00:00Z" + }, + "Image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", + "ResolvConfPath": "/var/lib/docker/containers/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127/resolv.conf", + "HostnamePath": "/var/lib/docker/containers/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127/hostname", + "HostsPath": "/var/lib/docker/containers/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127/hosts", + "LogPath": "/var/lib/docker/containers/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127/e4e0a92af9b8ba16bd3c139e9e3d04e074af2dff3a545a7ea5245b6c94c17127-json.log", + "Name": "/qwen38-a-hr", + "RestartCount": 0, + "Driver": "overlayfs", + "Platform": "linux", + "MountLabel": "", + "ProcessLabel": "", + "AppArmorProfile": "docker-default", + "ExecIDs": null, + "HostConfig": { + "Binds": [ + "/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", + "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro", + "/dev/dri:/dev/dri", + "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", + "/home/devcloud/hicache-shared:/hicache", + "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", + "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", + "/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", + "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro" + ], + "ContainerIDFile": "", + "LogConfig": { + "Type": "json-file", + "Config": {} + }, + "NetworkMode": "bridge", + "PortBindings": { + "30000/tcp": [ + { + "HostIp": "", + "HostPort": "30000" + } + ], + "5557/tcp": [ + { + "HostIp": "", + "HostPort": "5557" + } + ] + }, + "RestartPolicy": { + "Name": "no", + "MaximumRetryCount": 0 + }, + "AutoRemove": false, + "VolumeDriver": "", + "VolumesFrom": null, + "ConsoleSize": [ + 0, + 0 + ], + "CapAdd": null, + "CapDrop": null, + "CgroupnsMode": "private", + "Dns": null, + "DnsOptions": [], + "DnsSearch": [], + "ExtraHosts": null, + "GroupAdd": null, + "IpcMode": "host", + "Cgroup": "", + "Links": null, + "OomScoreAdj": 0, + "PidMode": "", + "Privileged": false, + "PublishAllPorts": false, + "ReadonlyRootfs": false, + "SecurityOpt": [ + "label=disable" + ], + "UTSMode": "", + "UsernsMode": "", + "ShmSize": 34359738368, + "Runtime": "runc", + "Isolation": "", + "CpuShares": 0, + "Memory": 0, + "NanoCpus": 0, + "CgroupParent": "", + "BlkioWeight": 0, + "BlkioWeightDevice": [], + "BlkioDeviceReadBps": [], + "BlkioDeviceWriteBps": [], + "BlkioDeviceReadIOps": [], + "BlkioDeviceWriteIOps": [], + "CpuPeriod": 0, + "CpuQuota": 0, + "CpuRealtimePeriod": 0, + "CpuRealtimeRuntime": 0, + "CpusetCpus": "", + "CpusetMems": "", + "Devices": [ + { + "PathOnHost": "/dev/dri", + "PathInContainer": "/dev/dri", + "CgroupPermissions": "rwm" + } + ], + "DeviceCgroupRules": null, + "DeviceRequests": null, + "MemoryReservation": 0, + "MemorySwap": 0, + "MemorySwappiness": null, + "OomKillDisable": null, + "PidsLimit": null, + "Ulimits": null, + "CpuCount": 0, + "CpuPercent": 0, + "IOMaximumIOps": 0, + "IOMaximumBandwidth": 0, + "MaskedPaths": [ + "/proc/acpi", + "/proc/asound", + "/proc/interrupts", + "/proc/kcore", + "/proc/keys", + "/proc/latency_stats", + "/proc/sched_debug", + "/proc/scsi", + "/proc/timer_list", + "/proc/timer_stats", + "/sys/devices/virtual/powercap", + "/sys/firmware", + "/sys/devices/system/cpu/cpu0/thermal_throttle", + "/sys/devices/system/cpu/cpu1/thermal_throttle", + "/sys/devices/system/cpu/cpu2/thermal_throttle", + "/sys/devices/system/cpu/cpu3/thermal_throttle", + "/sys/devices/system/cpu/cpu4/thermal_throttle", + "/sys/devices/system/cpu/cpu5/thermal_throttle", + "/sys/devices/system/cpu/cpu6/thermal_throttle", + "/sys/devices/system/cpu/cpu7/thermal_throttle", + "/sys/devices/system/cpu/cpu8/thermal_throttle", + "/sys/devices/system/cpu/cpu9/thermal_throttle", + "/sys/devices/system/cpu/cpu10/thermal_throttle", + "/sys/devices/system/cpu/cpu11/thermal_throttle", + "/sys/devices/system/cpu/cpu12/thermal_throttle", + "/sys/devices/system/cpu/cpu13/thermal_throttle", + "/sys/devices/system/cpu/cpu14/thermal_throttle", + "/sys/devices/system/cpu/cpu15/thermal_throttle", + "/sys/devices/system/cpu/cpu16/thermal_throttle", + "/sys/devices/system/cpu/cpu17/thermal_throttle", + "/sys/devices/system/cpu/cpu18/thermal_throttle", + "/sys/devices/system/cpu/cpu19/thermal_throttle", + "/sys/devices/system/cpu/cpu20/thermal_throttle", + "/sys/devices/system/cpu/cpu21/thermal_throttle", + "/sys/devices/system/cpu/cpu22/thermal_throttle", + "/sys/devices/system/cpu/cpu23/thermal_throttle", + "/sys/devices/system/cpu/cpu24/thermal_throttle", + "/sys/devices/system/cpu/cpu25/thermal_throttle", + "/sys/devices/system/cpu/cpu26/thermal_throttle", + "/sys/devices/system/cpu/cpu27/thermal_throttle", + "/sys/devices/system/cpu/cpu28/thermal_throttle", + "/sys/devices/system/cpu/cpu29/thermal_throttle", + "/sys/devices/system/cpu/cpu30/thermal_throttle", + "/sys/devices/system/cpu/cpu31/thermal_throttle", + "/sys/devices/system/cpu/cpu32/thermal_throttle", + "/sys/devices/system/cpu/cpu33/thermal_throttle", + "/sys/devices/system/cpu/cpu34/thermal_throttle", + "/sys/devices/system/cpu/cpu35/thermal_throttle", + "/sys/devices/system/cpu/cpu36/thermal_throttle", + "/sys/devices/system/cpu/cpu37/thermal_throttle", + "/sys/devices/system/cpu/cpu38/thermal_throttle", + "/sys/devices/system/cpu/cpu39/thermal_throttle", + "/sys/devices/system/cpu/cpu40/thermal_throttle", + "/sys/devices/system/cpu/cpu41/thermal_throttle", + "/sys/devices/system/cpu/cpu42/thermal_throttle", + "/sys/devices/system/cpu/cpu43/thermal_throttle", + "/sys/devices/system/cpu/cpu44/thermal_throttle", + "/sys/devices/system/cpu/cpu45/thermal_throttle", + "/sys/devices/system/cpu/cpu46/thermal_throttle", + "/sys/devices/system/cpu/cpu47/thermal_throttle", + "/sys/devices/system/cpu/cpu48/thermal_throttle", + "/sys/devices/system/cpu/cpu49/thermal_throttle", + "/sys/devices/system/cpu/cpu50/thermal_throttle", + "/sys/devices/system/cpu/cpu51/thermal_throttle", + "/sys/devices/system/cpu/cpu52/thermal_throttle", + "/sys/devices/system/cpu/cpu53/thermal_throttle", + "/sys/devices/system/cpu/cpu54/thermal_throttle", + "/sys/devices/system/cpu/cpu55/thermal_throttle", + "/sys/devices/system/cpu/cpu56/thermal_throttle", + "/sys/devices/system/cpu/cpu57/thermal_throttle", + "/sys/devices/system/cpu/cpu58/thermal_throttle", + "/sys/devices/system/cpu/cpu59/thermal_throttle", + "/sys/devices/system/cpu/cpu60/thermal_throttle", + "/sys/devices/system/cpu/cpu61/thermal_throttle", + "/sys/devices/system/cpu/cpu62/thermal_throttle", + "/sys/devices/system/cpu/cpu63/thermal_throttle", + "/sys/devices/system/cpu/cpu64/thermal_throttle", + "/sys/devices/system/cpu/cpu65/thermal_throttle", + "/sys/devices/system/cpu/cpu66/thermal_throttle", + "/sys/devices/system/cpu/cpu67/thermal_throttle", + "/sys/devices/system/cpu/cpu68/thermal_throttle", + "/sys/devices/system/cpu/cpu69/thermal_throttle", + "/sys/devices/system/cpu/cpu70/thermal_throttle", + "/sys/devices/system/cpu/cpu71/thermal_throttle", + "/sys/devices/system/cpu/cpu72/thermal_throttle", + "/sys/devices/system/cpu/cpu73/thermal_throttle", + "/sys/devices/system/cpu/cpu74/thermal_throttle", + "/sys/devices/system/cpu/cpu75/thermal_throttle", + "/sys/devices/system/cpu/cpu76/thermal_throttle", + "/sys/devices/system/cpu/cpu77/thermal_throttle", + "/sys/devices/system/cpu/cpu78/thermal_throttle", + "/sys/devices/system/cpu/cpu79/thermal_throttle", + "/sys/devices/system/cpu/cpu80/thermal_throttle", + "/sys/devices/system/cpu/cpu81/thermal_throttle", + "/sys/devices/system/cpu/cpu82/thermal_throttle", + "/sys/devices/system/cpu/cpu83/thermal_throttle", + "/sys/devices/system/cpu/cpu84/thermal_throttle", + "/sys/devices/system/cpu/cpu85/thermal_throttle", + "/sys/devices/system/cpu/cpu86/thermal_throttle", + "/sys/devices/system/cpu/cpu87/thermal_throttle", + "/sys/devices/system/cpu/cpu88/thermal_throttle", + "/sys/devices/system/cpu/cpu89/thermal_throttle", + "/sys/devices/system/cpu/cpu90/thermal_throttle", + "/sys/devices/system/cpu/cpu91/thermal_throttle", + "/sys/devices/system/cpu/cpu92/thermal_throttle", + "/sys/devices/system/cpu/cpu93/thermal_throttle", + "/sys/devices/system/cpu/cpu94/thermal_throttle", + "/sys/devices/system/cpu/cpu95/thermal_throttle" + ], + "ReadonlyPaths": [ + "/proc/bus", + "/proc/fs", + "/proc/irq", + "/proc/sys", + "/proc/sysrq-trigger" + ] + }, + "Storage": { + "RootFS": { + "Snapshot": { + "Name": "overlayfs" + } + } + }, + "Mounts": [ + { + "Type": "bind", + "Source": "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh", + "Destination": "/entry.sh", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/hicache-shared", + "Destination": "/hicache", + "Mode": "", + "RW": true, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4", + "Destination": "/model", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/dev/dri", + "Destination": "/dev/dri", + "Mode": "", + "RW": true, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/empty-memcache-pyc", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-mph.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/empty-pycache", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-common.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-mha.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + } + ], + "Config": { + "Hostname": "e4e0a92af9b8", + "Domainname": "", + "User": "root", + "AttachStdin": false, + "AttachStdout": false, + "AttachStderr": false, + "ExposedPorts": { + "30000/tcp": {}, + "5557/tcp": {} + }, + "Tty": false, + "OpenStdin": false, + "StdinOnce": false, + "Env": [ + "XPU_SYMM_AR=1", + "KV_PORT=5557", + "TP_SIZE=2", + "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", + "XPU_MAMBA_PAGED=1", + "XPU_DENSE_INT8_MAX_M=8", + "XPU_SYMM_AR_MAX_BYTES=131072", + "XPU_UNROLL_SPEC_VERIFY=1", + "PORT=30000", + "CTX=8192", + "XPU_DENSE_INT8=1", + "ZE_AFFINITY_MASK=0,1", + "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", + "XPU_GDN_DECODE_BV=16", + "XPU_MOE_ALIGN_SKIP=1", + "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "LANG=C.UTF-8", + "CCL_CONFIGURATION=cpu_gpu_dpcpp", + "CCL_CONFIGURATION_PATH=", + "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", + "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", + "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", + "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", + "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", + "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", + "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", + "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", + "DPLROOT=/opt/intel/oneapi/dpl/2022.13", + "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", + "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", + "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", + "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", + "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", + "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", + "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", + "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", + "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", + "MKLROOT=/opt/intel/oneapi/mkl/2026.1", + "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", + "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", + "ONEAPI_ROOT=/opt/intel/oneapi", + "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", + "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", + "SETVARS_COMPLETED=1", + "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", + "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", + "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", + "UMF_ROOT=/opt/intel/oneapi/umf/1.1", + "DEBIAN_FRONTEND=noninteractive", + "VIRTUAL_ENV=/opt/venv", + "UV_PYTHON_INSTALL_DIR=/opt/uv/python", + "XPU_TOPK_BUILD=/opt/xpu/build", + "XPU_MOE_BUILD=/opt/xpu/build-gemv", + "XPU_REPO=/opt/xpu", + "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", + "XPU_KERNELS=topk,gemv", + "SGLANG_TP_SIZE=2", + "SGLANG_HTTP_PORT=30000", + "SGLANG_CONTEXT_LEN=262144", + "SGLANG_MEM_FRACTION=0.85", + "SGLANG_CHUNKED_PREFILL=4096", + "SGLANG_MAX_RUNNING=64", + "SGLANG_PAGE_SIZE=64", + "SGLANG_MODEL=/model", + "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1" + ], + "Cmd": [ + "/entry.sh" + ], + "Image": "rahulunair/sglang-xpu@sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", + "Volumes": null, + "WorkingDir": "/sgl-workspace/sglang", + "Entrypoint": [ + "bash" + ], + "Labels": { + "org.opencontainers.image.description": "SGLang serving Qwen3.8-27B AWQ-INT4 or FP8 on four Intel Arc Pro B70 GPUs, with MTP and DFlash2 speculative decoding through the checkpoint's native 262,144-token context.", + "org.opencontainers.image.title": "Qwen3.8-27B on Intel Arc Pro", + "org.opencontainers.image.version": "24.04", + "xpu.qwen38.base": "rahulunair/sglang-xpu:qwen3.8-27b-20260819", + "xpu.qwen38.built": "20260820", + "xpu.qwen38.dflash2_source": "e5a3e4d30fa7abda95bafd2d697f9f9c48566114" + } + }, + "NetworkSettings": { + "SandboxID": "0a87eb278cbd20310832479bc36cefc9c3ff495719de11a71f74bf1a2d2b212d", + "SandboxKey": "/var/run/docker/netns/0a87eb278cbd", + "Ports": { + "30000/tcp": [ + { + "HostIp": "0.0.0.0", + "HostPort": "30000" + }, + { + "HostIp": "::", + "HostPort": "30000" + } + ], + "5557/tcp": [ + { + "HostIp": "0.0.0.0", + "HostPort": "5557" + }, + { + "HostIp": "::", + "HostPort": "5557" + } + ] + }, + "Networks": { + "bridge": { + "IPAMConfig": null, + "Links": null, + "Aliases": null, + "DriverOpts": null, + "GwPriority": 0, + "NetworkID": "079840671febd9bc1caa55e83e63558724adbe92299b1abb5777fe9fa72b5975", + "EndpointID": "2633a049bbcd307089aa4f67b15ce5962783ed209543218d550d6c1e760462fd", + "Gateway": "172.17.0.1", + "IPAddress": "172.17.0.2", + "MacAddress": "6a:23:3b:de:2a:39", + "IPPrefixLen": 16, + "IPv6Gateway": "", + "GlobalIPv6Address": "", + "GlobalIPv6PrefixLen": 0, + "DNSNames": null + } + } + }, + "ImageManifestDescriptor": { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "digest": "sha256:70aab3fb164f7941db3ec1b45be934bad70ff720f4013fd5b4429b3d90668e5a", + "size": 5653, + "platform": { + "architecture": "amd64", + "os": "linux" + } + } + } +] diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.key.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.key.json new file mode 100644 index 00000000..63bd34fe --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-a-hr.before.key.json @@ -0,0 +1 @@ +{"args": ["/entry.sh"], "binds": ["/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro", "/dev/dri:/dev/dri", "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", "/home/devcloud/hicache-shared:/hicache", "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", "/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro"], "devices": [{"CgroupPermissions": "rwm", "PathInContainer": "/dev/dri", "PathOnHost": "/dev/dri"}], "env": ["CCL_CONFIGURATION=cpu_gpu_dpcpp", "CCL_CONFIGURATION_PATH=", "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", "CTX=8192", "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", "DEBIAN_FRONTEND=noninteractive", "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", "DPLROOT=/opt/intel/oneapi/dpl/2022.13", "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", "KV_PORT=5557", "LANG=C.UTF-8", "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", "MKLROOT=/opt/intel/oneapi/mkl/2026.1", "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", "ONEAPI_ROOT=/opt/intel/oneapi", "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", "PORT=30000", "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", "SETVARS_COMPLETED=1", "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", "SGLANG_CHUNKED_PREFILL=4096", "SGLANG_CONTEXT_LEN=262144", "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", "SGLANG_HTTP_PORT=30000", "SGLANG_MAX_RUNNING=64", "SGLANG_MEM_FRACTION=0.85", "SGLANG_MODEL=/model", "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", "SGLANG_PAGE_SIZE=64", "SGLANG_TP_SIZE=2", "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", "TP_SIZE=2", "UMF_ROOT=/opt/intel/oneapi/umf/1.1", "UV_PYTHON_INSTALL_DIR=/opt/uv/python", "VIRTUAL_ENV=/opt/venv", "XPU_DENSE_INT8=1", "XPU_DENSE_INT8_MAX_M=8", "XPU_GDN_DECODE_BV=16", "XPU_KERNELS=topk,gemv", "XPU_MAMBA_PAGED=1", "XPU_MOE_ALIGN_SKIP=1", "XPU_MOE_BUILD=/opt/xpu/build-gemv", "XPU_REPO=/opt/xpu", "XPU_SYMM_AR=1", "XPU_SYMM_AR_MAX_BYTES=131072", "XPU_TOPK_BUILD=/opt/xpu/build", "XPU_UNROLL_SPEC_VERIFY=1", "ZE_AFFINITY_MASK=0,1"], "image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", "network": "bridge", "path": "bash", "ports": {"30000/tcp": [{"HostIp": "", "HostPort": "30000"}], "5557/tcp": [{"HostIp": "", "HostPort": "5557"}]}, "restart": {"MaximumRetryCount": 0, "Name": "no"}} diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.after.key.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.after.key.json new file mode 100644 index 00000000..c8dd0d9d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.after.key.json @@ -0,0 +1 @@ +{"args": ["/entry.sh"], "binds": ["/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", "/dev/dri:/dev/dri", "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", "/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro", "/home/devcloud/hicache-shared:/hicache", "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro"], "devices": [{"CgroupPermissions": "rwm", "PathInContainer": "/dev/dri", "PathOnHost": "/dev/dri"}], "env": ["CCL_CONFIGURATION=cpu_gpu_dpcpp", "CCL_CONFIGURATION_PATH=", "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", "CTX=8192", "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", "DEBIAN_FRONTEND=noninteractive", "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", "DPLROOT=/opt/intel/oneapi/dpl/2022.13", "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", "KV_PORT=5558", "LANG=C.UTF-8", "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", "MKLROOT=/opt/intel/oneapi/mkl/2026.1", "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", "ONEAPI_ROOT=/opt/intel/oneapi", "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", "PORT=30001", "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", "SETVARS_COMPLETED=1", "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", "SGLANG_CHUNKED_PREFILL=4096", "SGLANG_CONTEXT_LEN=262144", "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", "SGLANG_HTTP_PORT=30000", "SGLANG_MAX_RUNNING=64", "SGLANG_MEM_FRACTION=0.85", "SGLANG_MODEL=/model", "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", "SGLANG_PAGE_SIZE=64", "SGLANG_TP_SIZE=2", "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", "TP_SIZE=2", "UMF_ROOT=/opt/intel/oneapi/umf/1.1", "UV_PYTHON_INSTALL_DIR=/opt/uv/python", "VIRTUAL_ENV=/opt/venv", "XPU_DENSE_INT8=1", "XPU_DENSE_INT8_MAX_M=8", "XPU_GDN_DECODE_BV=16", "XPU_KERNELS=topk,gemv", "XPU_MAMBA_PAGED=1", "XPU_MOE_ALIGN_SKIP=1", "XPU_MOE_BUILD=/opt/xpu/build-gemv", "XPU_REPO=/opt/xpu", "XPU_SYMM_AR=1", "XPU_SYMM_AR_MAX_BYTES=131072", "XPU_TOPK_BUILD=/opt/xpu/build", "XPU_UNROLL_SPEC_VERIFY=1", "ZE_AFFINITY_MASK=2,3"], "image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", "network": "bridge", "path": "bash", "ports": {"30001/tcp": [{"HostIp": "", "HostPort": "30001"}], "5558/tcp": [{"HostIp": "", "HostPort": "5558"}]}, "restart": {"MaximumRetryCount": 0, "Name": "no"}} diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.json new file mode 100644 index 00000000..42aeb1bd --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.json @@ -0,0 +1,493 @@ +[ + { + "Id": "e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80", + "Created": "2026-10-19T07:43:07.195804765Z", + "Path": "bash", + "Args": [ + "/entry.sh" + ], + "State": { + "Status": "running", + "Running": true, + "Paused": false, + "Restarting": false, + "OOMKilled": false, + "Dead": false, + "Pid": 528967, + "ExitCode": 0, + "Error": "", + "StartedAt": "2026-10-21T21:57:40.048150052Z", + "FinishedAt": "2026-10-21T21:12:31.94913436Z" + }, + "Image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", + "ResolvConfPath": "/var/lib/docker/containers/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80/resolv.conf", + "HostnamePath": "/var/lib/docker/containers/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80/hostname", + "HostsPath": "/var/lib/docker/containers/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80/hosts", + "LogPath": "/var/lib/docker/containers/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80/e4d6195699d9fefd2306b782974274c4b4c3cd52d6a50b596f2c726f6a50cb80-json.log", + "Name": "/qwen38-b-hr", + "RestartCount": 0, + "Driver": "overlayfs", + "Platform": "linux", + "MountLabel": "", + "ProcessLabel": "", + "AppArmorProfile": "docker-default", + "ExecIDs": null, + "HostConfig": { + "Binds": [ + "/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", + "/dev/dri:/dev/dri", + "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", + "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", + "/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", + "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro", + "/home/devcloud/hicache-shared:/hicache", + "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", + "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro" + ], + "ContainerIDFile": "", + "LogConfig": { + "Type": "json-file", + "Config": {} + }, + "NetworkMode": "bridge", + "PortBindings": { + "30001/tcp": [ + { + "HostIp": "", + "HostPort": "30001" + } + ], + "5558/tcp": [ + { + "HostIp": "", + "HostPort": "5558" + } + ] + }, + "RestartPolicy": { + "Name": "no", + "MaximumRetryCount": 0 + }, + "AutoRemove": false, + "VolumeDriver": "", + "VolumesFrom": null, + "ConsoleSize": [ + 0, + 0 + ], + "CapAdd": null, + "CapDrop": null, + "CgroupnsMode": "private", + "Dns": null, + "DnsOptions": [], + "DnsSearch": [], + "ExtraHosts": null, + "GroupAdd": null, + "IpcMode": "host", + "Cgroup": "", + "Links": null, + "OomScoreAdj": 0, + "PidMode": "", + "Privileged": false, + "PublishAllPorts": false, + "ReadonlyRootfs": false, + "SecurityOpt": [ + "label=disable" + ], + "UTSMode": "", + "UsernsMode": "", + "ShmSize": 34359738368, + "Runtime": "runc", + "Isolation": "", + "CpuShares": 0, + "Memory": 0, + "NanoCpus": 0, + "CgroupParent": "", + "BlkioWeight": 0, + "BlkioWeightDevice": [], + "BlkioDeviceReadBps": [], + "BlkioDeviceWriteBps": [], + "BlkioDeviceReadIOps": [], + "BlkioDeviceWriteIOps": [], + "CpuPeriod": 0, + "CpuQuota": 0, + "CpuRealtimePeriod": 0, + "CpuRealtimeRuntime": 0, + "CpusetCpus": "", + "CpusetMems": "", + "Devices": [ + { + "PathOnHost": "/dev/dri", + "PathInContainer": "/dev/dri", + "CgroupPermissions": "rwm" + } + ], + "DeviceCgroupRules": null, + "DeviceRequests": null, + "MemoryReservation": 0, + "MemorySwap": 0, + "MemorySwappiness": null, + "OomKillDisable": null, + "PidsLimit": null, + "Ulimits": null, + "CpuCount": 0, + "CpuPercent": 0, + "IOMaximumIOps": 0, + "IOMaximumBandwidth": 0, + "MaskedPaths": [ + "/proc/acpi", + "/proc/asound", + "/proc/interrupts", + "/proc/kcore", + "/proc/keys", + "/proc/latency_stats", + "/proc/sched_debug", + "/proc/scsi", + "/proc/timer_list", + "/proc/timer_stats", + "/sys/devices/virtual/powercap", + "/sys/firmware", + "/sys/devices/system/cpu/cpu0/thermal_throttle", + "/sys/devices/system/cpu/cpu1/thermal_throttle", + "/sys/devices/system/cpu/cpu2/thermal_throttle", + "/sys/devices/system/cpu/cpu3/thermal_throttle", + "/sys/devices/system/cpu/cpu4/thermal_throttle", + "/sys/devices/system/cpu/cpu5/thermal_throttle", + "/sys/devices/system/cpu/cpu6/thermal_throttle", + "/sys/devices/system/cpu/cpu7/thermal_throttle", + "/sys/devices/system/cpu/cpu8/thermal_throttle", + "/sys/devices/system/cpu/cpu9/thermal_throttle", + "/sys/devices/system/cpu/cpu10/thermal_throttle", + "/sys/devices/system/cpu/cpu11/thermal_throttle", + "/sys/devices/system/cpu/cpu12/thermal_throttle", + "/sys/devices/system/cpu/cpu13/thermal_throttle", + "/sys/devices/system/cpu/cpu14/thermal_throttle", + "/sys/devices/system/cpu/cpu15/thermal_throttle", + "/sys/devices/system/cpu/cpu16/thermal_throttle", + "/sys/devices/system/cpu/cpu17/thermal_throttle", + "/sys/devices/system/cpu/cpu18/thermal_throttle", + "/sys/devices/system/cpu/cpu19/thermal_throttle", + "/sys/devices/system/cpu/cpu20/thermal_throttle", + "/sys/devices/system/cpu/cpu21/thermal_throttle", + "/sys/devices/system/cpu/cpu22/thermal_throttle", + "/sys/devices/system/cpu/cpu23/thermal_throttle", + "/sys/devices/system/cpu/cpu24/thermal_throttle", + "/sys/devices/system/cpu/cpu25/thermal_throttle", + "/sys/devices/system/cpu/cpu26/thermal_throttle", + "/sys/devices/system/cpu/cpu27/thermal_throttle", + "/sys/devices/system/cpu/cpu28/thermal_throttle", + "/sys/devices/system/cpu/cpu29/thermal_throttle", + "/sys/devices/system/cpu/cpu30/thermal_throttle", + "/sys/devices/system/cpu/cpu31/thermal_throttle", + "/sys/devices/system/cpu/cpu32/thermal_throttle", + "/sys/devices/system/cpu/cpu33/thermal_throttle", + "/sys/devices/system/cpu/cpu34/thermal_throttle", + "/sys/devices/system/cpu/cpu35/thermal_throttle", + "/sys/devices/system/cpu/cpu36/thermal_throttle", + "/sys/devices/system/cpu/cpu37/thermal_throttle", + "/sys/devices/system/cpu/cpu38/thermal_throttle", + "/sys/devices/system/cpu/cpu39/thermal_throttle", + "/sys/devices/system/cpu/cpu40/thermal_throttle", + "/sys/devices/system/cpu/cpu41/thermal_throttle", + "/sys/devices/system/cpu/cpu42/thermal_throttle", + "/sys/devices/system/cpu/cpu43/thermal_throttle", + "/sys/devices/system/cpu/cpu44/thermal_throttle", + "/sys/devices/system/cpu/cpu45/thermal_throttle", + "/sys/devices/system/cpu/cpu46/thermal_throttle", + "/sys/devices/system/cpu/cpu47/thermal_throttle", + "/sys/devices/system/cpu/cpu48/thermal_throttle", + "/sys/devices/system/cpu/cpu49/thermal_throttle", + "/sys/devices/system/cpu/cpu50/thermal_throttle", + "/sys/devices/system/cpu/cpu51/thermal_throttle", + "/sys/devices/system/cpu/cpu52/thermal_throttle", + "/sys/devices/system/cpu/cpu53/thermal_throttle", + "/sys/devices/system/cpu/cpu54/thermal_throttle", + "/sys/devices/system/cpu/cpu55/thermal_throttle", + "/sys/devices/system/cpu/cpu56/thermal_throttle", + "/sys/devices/system/cpu/cpu57/thermal_throttle", + "/sys/devices/system/cpu/cpu58/thermal_throttle", + "/sys/devices/system/cpu/cpu59/thermal_throttle", + "/sys/devices/system/cpu/cpu60/thermal_throttle", + "/sys/devices/system/cpu/cpu61/thermal_throttle", + "/sys/devices/system/cpu/cpu62/thermal_throttle", + "/sys/devices/system/cpu/cpu63/thermal_throttle", + "/sys/devices/system/cpu/cpu64/thermal_throttle", + "/sys/devices/system/cpu/cpu65/thermal_throttle", + "/sys/devices/system/cpu/cpu66/thermal_throttle", + "/sys/devices/system/cpu/cpu67/thermal_throttle", + "/sys/devices/system/cpu/cpu68/thermal_throttle", + "/sys/devices/system/cpu/cpu69/thermal_throttle", + "/sys/devices/system/cpu/cpu70/thermal_throttle", + "/sys/devices/system/cpu/cpu71/thermal_throttle", + "/sys/devices/system/cpu/cpu72/thermal_throttle", + "/sys/devices/system/cpu/cpu73/thermal_throttle", + "/sys/devices/system/cpu/cpu74/thermal_throttle", + "/sys/devices/system/cpu/cpu75/thermal_throttle", + "/sys/devices/system/cpu/cpu76/thermal_throttle", + "/sys/devices/system/cpu/cpu77/thermal_throttle", + "/sys/devices/system/cpu/cpu78/thermal_throttle", + "/sys/devices/system/cpu/cpu79/thermal_throttle", + "/sys/devices/system/cpu/cpu80/thermal_throttle", + "/sys/devices/system/cpu/cpu81/thermal_throttle", + "/sys/devices/system/cpu/cpu82/thermal_throttle", + "/sys/devices/system/cpu/cpu83/thermal_throttle", + "/sys/devices/system/cpu/cpu84/thermal_throttle", + "/sys/devices/system/cpu/cpu85/thermal_throttle", + "/sys/devices/system/cpu/cpu86/thermal_throttle", + "/sys/devices/system/cpu/cpu87/thermal_throttle", + "/sys/devices/system/cpu/cpu88/thermal_throttle", + "/sys/devices/system/cpu/cpu89/thermal_throttle", + "/sys/devices/system/cpu/cpu90/thermal_throttle", + "/sys/devices/system/cpu/cpu91/thermal_throttle", + "/sys/devices/system/cpu/cpu92/thermal_throttle", + "/sys/devices/system/cpu/cpu93/thermal_throttle", + "/sys/devices/system/cpu/cpu94/thermal_throttle", + "/sys/devices/system/cpu/cpu95/thermal_throttle" + ], + "ReadonlyPaths": [ + "/proc/bus", + "/proc/fs", + "/proc/irq", + "/proc/sys", + "/proc/sysrq-trigger" + ] + }, + "Storage": { + "RootFS": { + "Snapshot": { + "Name": "overlayfs" + } + } + }, + "Mounts": [ + { + "Type": "bind", + "Source": "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh", + "Destination": "/entry.sh", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/hicache-shared", + "Destination": "/hicache", + "Mode": "", + "RW": true, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4", + "Destination": "/model", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/dev/dri", + "Destination": "/dev/dri", + "Mode": "", + "RW": true, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/empty-memcache-pyc", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-mph.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/empty-pycache", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-common.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + }, + { + "Type": "bind", + "Source": "/home/devcloud/launchers/hicache-mha.py", + "Destination": "/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py", + "Mode": "ro", + "RW": false, + "Propagation": "rprivate" + } + ], + "Config": { + "Hostname": "e4d6195699d9", + "Domainname": "", + "User": "root", + "AttachStdin": false, + "AttachStdout": false, + "AttachStderr": false, + "ExposedPorts": { + "30000/tcp": {}, + "30001/tcp": {}, + "5558/tcp": {} + }, + "Tty": false, + "OpenStdin": false, + "StdinOnce": false, + "Env": [ + "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", + "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", + "XPU_DENSE_INT8=1", + "XPU_SYMM_AR=1", + "XPU_MAMBA_PAGED=1", + "XPU_DENSE_INT8_MAX_M=8", + "CTX=8192", + "KV_PORT=5558", + "ZE_AFFINITY_MASK=2,3", + "XPU_SYMM_AR_MAX_BYTES=131072", + "XPU_UNROLL_SPEC_VERIFY=1", + "XPU_GDN_DECODE_BV=16", + "XPU_MOE_ALIGN_SKIP=1", + "PORT=30001", + "TP_SIZE=2", + "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "LANG=C.UTF-8", + "CCL_CONFIGURATION=cpu_gpu_dpcpp", + "CCL_CONFIGURATION_PATH=", + "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", + "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", + "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", + "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", + "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", + "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", + "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", + "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", + "DPLROOT=/opt/intel/oneapi/dpl/2022.13", + "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", + "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", + "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", + "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", + "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", + "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", + "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", + "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", + "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", + "MKLROOT=/opt/intel/oneapi/mkl/2026.1", + "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", + "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", + "ONEAPI_ROOT=/opt/intel/oneapi", + "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", + "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", + "SETVARS_COMPLETED=1", + "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", + "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", + "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", + "UMF_ROOT=/opt/intel/oneapi/umf/1.1", + "DEBIAN_FRONTEND=noninteractive", + "VIRTUAL_ENV=/opt/venv", + "UV_PYTHON_INSTALL_DIR=/opt/uv/python", + "XPU_TOPK_BUILD=/opt/xpu/build", + "XPU_MOE_BUILD=/opt/xpu/build-gemv", + "XPU_REPO=/opt/xpu", + "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", + "XPU_KERNELS=topk,gemv", + "SGLANG_TP_SIZE=2", + "SGLANG_HTTP_PORT=30000", + "SGLANG_CONTEXT_LEN=262144", + "SGLANG_MEM_FRACTION=0.85", + "SGLANG_CHUNKED_PREFILL=4096", + "SGLANG_MAX_RUNNING=64", + "SGLANG_PAGE_SIZE=64", + "SGLANG_MODEL=/model", + "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", + "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1" + ], + "Cmd": [ + "/entry.sh" + ], + "Image": "rahulunair/sglang-xpu@sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", + "Volumes": null, + "WorkingDir": "/sgl-workspace/sglang", + "Entrypoint": [ + "bash" + ], + "Labels": { + "org.opencontainers.image.description": "SGLang serving Qwen3.8-27B AWQ-INT4 or FP8 on four Intel Arc Pro B70 GPUs, with MTP and DFlash2 speculative decoding through the checkpoint's native 262,144-token context.", + "org.opencontainers.image.title": "Qwen3.8-27B on Intel Arc Pro", + "org.opencontainers.image.version": "24.04", + "xpu.qwen38.base": "rahulunair/sglang-xpu:qwen3.8-27b-20260819", + "xpu.qwen38.built": "20260820", + "xpu.qwen38.dflash2_source": "e5a3e4d30fa7abda95bafd2d697f9f9c48566114" + } + }, + "NetworkSettings": { + "SandboxID": "03f3987afe75b41bf575db9f24465c03eadd18d0db38c1159b382d341eaa41a2", + "SandboxKey": "/var/run/docker/netns/03f3987afe75", + "Ports": { + "30000/tcp": null, + "30001/tcp": [ + { + "HostIp": "0.0.0.0", + "HostPort": "30001" + }, + { + "HostIp": "::", + "HostPort": "30001" + } + ], + "5558/tcp": [ + { + "HostIp": "0.0.0.0", + "HostPort": "5558" + }, + { + "HostIp": "::", + "HostPort": "5558" + } + ] + }, + "Networks": { + "bridge": { + "IPAMConfig": null, + "Links": null, + "Aliases": null, + "DriverOpts": null, + "GwPriority": 0, + "NetworkID": "079840671febd9bc1caa55e83e63558724adbe92299b1abb5777fe9fa72b5975", + "EndpointID": "ffa4f82708a4cfbd1ea62d5979fb28595208835dc0b1b2922cd2fde66b611b75", + "Gateway": "172.17.0.1", + "IPAddress": "172.17.0.3", + "MacAddress": "5a:ad:91:4a:45:d3", + "IPPrefixLen": 16, + "IPv6Gateway": "", + "GlobalIPv6Address": "", + "GlobalIPv6PrefixLen": 0, + "DNSNames": null + } + } + }, + "ImageManifestDescriptor": { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "digest": "sha256:70aab3fb164f7941db3ec1b45be934bad70ff720f4013fd5b4429b3d90668e5a", + "size": 5653, + "platform": { + "architecture": "amd64", + "os": "linux" + } + } + } +] diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.key.json b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.key.json new file mode 100644 index 00000000..c8dd0d9d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/inspect-qwen38-b-hr.before.key.json @@ -0,0 +1 @@ +{"args": ["/entry.sh"], "binds": ["/home/devcloud/launchers/empty-memcache-pyc:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/__pycache__:ro", "/dev/dri:/dev/dri", "/home/devcloud/launchers/qwen38-pair-entry-hicache.sh:/entry.sh:ro", "/home/devcloud/launchers/hicache-common.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/common.py:ro", "/home/devcloud/launchers/hicache-mha.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/mha.py:ro", "/home/devcloud/models/Qwen3.8-27B-AWQ-INT4:/model:ro", "/home/devcloud/hicache-shared:/hicache", "/home/devcloud/launchers/hicache-mph.py:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/memory_pool_host.py:ro", "/home/devcloud/launchers/empty-pycache:/opt/venv/lib/python3.12/site-packages/sglang/srt/mem_cache/pool_host/__pycache__:ro"], "devices": [{"CgroupPermissions": "rwm", "PathInContainer": "/dev/dri", "PathOnHost": "/dev/dri"}], "env": ["CCL_CONFIGURATION=cpu_gpu_dpcpp", "CCL_CONFIGURATION_PATH=", "CCL_ROOT=/opt/intel/oneapi/ccl/2022.1", "CLASSPATH=/opt/intel/oneapi/mpi/2021.18/share/java/mpi.jar", "CMAKE_PREFIX_PATH=/opt/intel/oneapi/tbb/2023.1/env/..:/opt/intel/oneapi/pti/1.0/lib/cmake/pti:/opt/intel/oneapi/mkl/2026.1/lib/cmake:/opt/intel/oneapi/dpl/2022.13/lib/cmake/oneDPL:/opt/intel/oneapi/compiler/2026.1:/opt/intel/oneapi/ccl/2022.1/lib/cmake/oneCCL", "CMPLR_ROOT=/opt/intel/oneapi/compiler/2026.1", "CPATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/dev-utilities/2026.0/include:/opt/intel/oneapi/ccl/2022.1/include", "CPLUS_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include:/opt/intel/oneapi/dpl/2022.13/include", "CTX=8192", "C_INCLUDE_PATH=/opt/intel/oneapi/umf/1.1/include:/opt/intel/oneapi/tbb/2023.1/env/../include:/opt/intel/oneapi/pti/1.0/include:/opt/intel/oneapi/mpi/2021.18/include:/opt/intel/oneapi/mkl/2026.1/include", "DEBIAN_FRONTEND=noninteractive", "DIAGUTIL_PATH=/opt/intel/oneapi/compiler/2026.1/etc/compiler/sys_check/sys_check.sh", "DPLROOT=/opt/intel/oneapi/dpl/2022.13", "DPL_ROOT=/opt/intel/oneapi/dpl/2022.13", "FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric", "GDB_INFO=/opt/intel/oneapi/debugger/2026.1/share/info/", "INFOPATH=/opt/intel/oneapi/debugger/2026.1/share/info", "INTEL_PYTHONHOME=/opt/intel/oneapi/debugger/2026.1/opt/debugger", "I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.18", "KV_PORT=5558", "LANG=C.UTF-8", "LD_LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tcm/1.5/env/../lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/debugger/2026.1/opt/debugger/lib:/opt/intel/oneapi/compiler/2026.1/opt/compiler/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "LIBRARY_PATH=/opt/intel/oneapi/tcm/1.5/lib:/opt/intel/oneapi/umf/1.1/lib:/opt/intel/oneapi/tbb/2023.1/env/../lib/intel64/gcc4.8:/opt/intel/oneapi/pti/1.0/lib:/opt/intel/oneapi/mpi/2021.18/lib:/opt/intel/oneapi/mkl/2026.1/lib:/opt/intel/oneapi/compiler/2026.1/lib:/opt/intel/oneapi/ccl/2022.1/lib/", "MANPATH=/opt/intel/oneapi/mpi/2021.18/share/man:/opt/intel/oneapi/debugger/2026.1/share/man:/opt/intel/oneapi/compiler/2026.1/share/man:", "MKLROOT=/opt/intel/oneapi/mkl/2026.1", "NLSPATH=/opt/intel/oneapi/compiler/2026.1/lib/compiler/locale/%l_%t/%N", "OCL_ICD_FILENAMES=/opt/intel/oneapi/compiler/2026.1/lib/libintelocl.so", "ONEAPI_DEVICE_SELECTOR=level_zero:gpu", "ONEAPI_ROOT=/opt/intel/oneapi", "PATH=/opt/venv/bin:/root/.local/bin:/opt/intel/oneapi/mpi/2021.18/bin:/opt/intel/oneapi/mkl/2026.1/bin:/opt/intel/oneapi/dev-utilities/2026.0/bin:/opt/intel/oneapi/debugger/2026.1/opt/debugger/bin:/opt/intel/oneapi/compiler/2026.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PKG_CONFIG_PATH=/opt/intel/oneapi/tbb/2023.1/env/../lib/pkgconfig:/opt/intel/oneapi/mpi/2021.18/lib/pkgconfig:/opt/intel/oneapi/mkl/2026.1/lib/pkgconfig:/opt/intel/oneapi/dpl/2022.13/lib/pkgconfig:/opt/intel/oneapi/compiler/2026.1/lib/pkgconfig:/opt/intel/oneapi/ccl/2022.1/lib/pkgconfig/", "PORT=30001", "Pti_DIR=/opt/intel/oneapi/pti/1.0/lib/cmake/pti", "SETVARS_COMPLETED=1", "SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1", "SGLANG_CHUNKED_PREFILL=4096", "SGLANG_CONTEXT_LEN=262144", "SGLANG_DFLASH_DISABLE_DRAFT_GRAPH=0", "SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/hicache", "SGLANG_HTTP_PORT=30000", "SGLANG_MAX_RUNNING=64", "SGLANG_MEM_FRACTION=0.85", "SGLANG_MODEL=/model", "SGLANG_MOE_CONFIG_DIR=/opt/xpu/moe-tuned", "SGLANG_PAGE_SIZE=64", "SGLANG_TP_SIZE=2", "TBBROOT=/opt/intel/oneapi/tbb/2023.1/env/..", "TCMROOT=/opt/intel/oneapi/tcm/1.5/env/..", "TCM_ROOT=/opt/intel/oneapi/tcm/1.5", "TP_SIZE=2", "UMF_ROOT=/opt/intel/oneapi/umf/1.1", "UV_PYTHON_INSTALL_DIR=/opt/uv/python", "VIRTUAL_ENV=/opt/venv", "XPU_DENSE_INT8=1", "XPU_DENSE_INT8_MAX_M=8", "XPU_GDN_DECODE_BV=16", "XPU_KERNELS=topk,gemv", "XPU_MAMBA_PAGED=1", "XPU_MOE_ALIGN_SKIP=1", "XPU_MOE_BUILD=/opt/xpu/build-gemv", "XPU_REPO=/opt/xpu", "XPU_SYMM_AR=1", "XPU_SYMM_AR_MAX_BYTES=131072", "XPU_TOPK_BUILD=/opt/xpu/build", "XPU_UNROLL_SPEC_VERIFY=1", "ZE_AFFINITY_MASK=2,3"], "image": "sha256:feb7b9130eff2fa26dfa09ab1a8b9a8423db013ad7d18f804f8b55a57cb2c175", "network": "bridge", "path": "bash", "ports": {"30001/tcp": [{"HostIp": "", "HostPort": "30001"}], "5558/tcp": [{"HostIp": "", "HostPort": "5558"}]}, "restart": {"MaximumRetryCount": 0, "Name": "no"}} diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/setup.txt b/bench/eatflow/results/2026-10-20/H53/prehold/setup.txt new file mode 100644 index 00000000..87e23150 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/setup.txt @@ -0,0 +1,16 @@ +lock-taken 2026-10-30T00:13:39Z +1f39488be7695dc0d1848347bf1c91f04025bc55 +k3d: OK +kubectl: OK +helm.tgz: OK +uv.tgz: OK +k3d version v5.9.0 +k3s version v1.35.5-k3s1 (default) +Client Version: v1.37.2 +v4.3.1+gc0d08d0 +uv 0.13.6 (x86_64-unknown-linux-gnu) +rc=0 2026-10-30T00:13:42Z +06d8f25bc3a971c4eb29e0ff08429b180402db0f4dec838c9eac427e296800a0 k3d +c04487523e7aa7d1493f7808174c899db93d7a9cf57531420cd76b9bd415f8f4 kubectl +b66cb19f2b31f9d427632ea6c4b4a1ff89b74ecaeaee323cd166da41a7af53bd helm +23803452c0fbe125033054cd9ff245f1d649fb65b4b73f68429f4abda432436d uv diff --git a/bench/eatflow/results/2026-10-20/H53/prehold/verify.txt b/bench/eatflow/results/2026-10-20/H53/prehold/verify.txt new file mode 100644 index 00000000..234aead3 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/prehold/verify.txt @@ -0,0 +1,64 @@ +lock-taken 2026-10-30T00:13:53Z +1f39488be7695dc0d1848347bf1c91f04025bc55 +ok size 1570 .gitattributes +ok size 11544 LICENSE +ok size 65012 README.md +ok size 8952 chat_template.jinja +ok size 4312 config.json +ok size 238 crc32.txt +ok size 202 generation_config.json +ok size 3353259 merges.txt +ok ba0ce20aae489ad196733da5064bcdf159a1fe84f53336648196e1ebb7751b1c model-00001-of-00018.safetensors +ok 06a148c01bfbe3faa14a5f184a7ff29a706f7ae1c8b2705d2058e26d17a001fb model-00002-of-00018.safetensors +ok 2e1bf62cbcd406eaa64b60d10353e1f0ef4039d0976e56f05cabe953454f9968 model-00003-of-00018.safetensors +ok 511e34063187882659753c4d93f3859f93c019fd438d8813071921c81d9a3f1a model-00004-of-00018.safetensors +ok 635cb53446dc74f219740fc59e18b774f877b803b9722e289ca62575a6efa701 model-00005-of-00018.safetensors +ok 0bc5214fac607f0e6cc92eec3789d4b8559410ef9fce66621ba8158e8410dae0 model-00006-of-00018.safetensors +ok 80b0c49033e9a0d5762562aa12f4acdb7f54da586f3d0110f28c48d91cf07892 model-00007-of-00018.safetensors +ok 7192c5b66185d3592927daabee1cc19e6f6e0ce75988ee20e824b624765fda79 model-00008-of-00018.safetensors +ok af3c48cc37af44f3db6ae0579baf019180d48d9c527caa0a1f03ff85813a56d8 model-00009-of-00018.safetensors +ok 163490a76f3bea3a40855b7efc04ce6d27afaf1a34f0bbde495b9491f76457c9 model-00010-of-00018.safetensors +ok 5f3ae1b948aeee39da77aec558e8236cd65fe4d7cb7686a76bb007acc563c6d8 model-00011-of-00018.safetensors +ok a3de1c7114677a8f5ac5c4892c90e8238ea5c1e2038c80e757dfc87c3902ca55 model-00012-of-00018.safetensors +ok 06ab79a41f74c9c5cb734816feb0c7fc364104b227165ee7391231e1155aa02a model-00013-of-00018.safetensors +ok 4138ed94603065ba884bbcadedb04d7718bb40117e85e6f5c6fc5b9c05b7a85b model-00014-of-00018.safetensors +ok 69224e27b9de4e7dbf6fc936c6eaae08447bda3b80a6c31a871ab451173afd22 model-00015-of-00018.safetensors +ok 73cb9a1089fb6155cb648609478d6633be8a5c7d9ca5a05bc8925ce8a553cefe model-00016-of-00018.safetensors +ok beb51f01056142ac4984bd800507b0dd0fd18de57f8e9ef6ea41d1a3598983a8 model-00017-of-00018.safetensors +ok 1d3479509e21494658f9b64d317f5ea8e55c4025d28c702d6c4d0b356ce8ea06 model-00018-of-00018.safetensors +ok size 112216 model.safetensors.index.json +ok size 390 preprocessor_config.json +ok 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3 tokenizer.json +ok size 17928 tokenizer_config.json +ok size 385 video_preprocessor_config.json +ok size 6722759 vocab.json +Qwen3.8-27B rc=0 +ok size 1570 .gitattributes +ok size 8321 README.md +ok size 8952 chat_template.jinja +ok size 3679 config.json +ok size 213 generation_config.json +ok size 1790 openvino_config.json +ok 48fc2c6a7a1056994982911eb9695826acef933c3d742364c56d16317a0b2a79 openvino_detokenizer.bin +ok size 16771 openvino_detokenizer.xml +ok 95aac72ac7286707c5e0fd9b120ea7f22c93f3ae17e7f1263442587d1d887d50 openvino_language_model.bin +ok size 9736672 openvino_language_model.xml +ok bc8ef47f252ad5a72be11e48ba6a23da50a67d1cef03396a7d902ca9aa1d5bc8 openvino_mtp_model.bin +ok size 210992 openvino_mtp_model.xml +ok e2099d0d83dcdfb00c378b51b50a52d28116aee71eb4f81cef0fe19ae37a085f openvino_text_embeddings_model.bin +ok size 5920 openvino_text_embeddings_model.xml +ok ca1ff37f580b558f0119b57824240adc92624d6d2ef6dd96ca2a24f4f1368c05 openvino_tokenizer.bin +ok size 35358 openvino_tokenizer.xml +ok 3792b6946c1f475aaac5d6f3960c7b912f612104e4f39ad68643b707370ed8e7 openvino_vision_embeddings_merger_model.bin +ok size 1330718 openvino_vision_embeddings_merger_model.xml +ok 5a8661b052abd07563d0f7148934d1fb77d5a842647151304287beb5dcee5541 openvino_vision_embeddings_model.bin +ok size 8610 openvino_vision_embeddings_model.xml +ok 82ccc97be0351cd82ebe1717fd7fa9adf72a44982a7ced8e8a845d869325d9e5 openvino_vision_embeddings_pos_model.bin +ok size 5887 openvino_vision_embeddings_pos_model.xml +ok size 390 preprocessor_config.json +ok size 1300 processor_config.json +ok 87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4 tokenizer.json +ok size 1098 tokenizer_config.json +Qwen3.8-27B-int4-ov rc=0 +listing equal to setup's: rc=0 +2026-10-30T00:14:39Z diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-post.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-post.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-pre.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/dmesg-pre.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/hold.log b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/hold.log new file mode 100644 index 00000000..5d3fe93c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/hold.log @@ -0,0 +1,41 @@ +== arm H53 start 00:15:14 (+0s) +Unable to find image 'python:3.14.8-slim@sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a' locally +docker.io/library/python@sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a: Pulling from library/python +82a60621211f: Pulling fs layer +f2be67225471: Pulling fs layer +77be27415a1b: Pulling fs layer +ac7730f4ce1f: Pulling fs layer +ac7730f4ce1f: Download complete +82a60621211f: Download complete +77be27415a1b: Download complete +c30c368ebede: Download complete +f2be67225471: Download complete +c3878e211f92: Download complete +f2be67225471: Pull complete +77be27415a1b: Pull complete +ac7730f4ce1f: Pull complete +82a60621211f: Pull complete +Digest: sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a +Status: Downloaded newer image for python@sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a +== k3d-gpu up 00:15:16 (+2s) +== wait-ready 00:18:24 (+190s) +Waiting for deployment "ovms" rollout to finish: 0 of 1 updated replicas are available... +deployment "ovms" successfully rolled out +== ready 00:19:15 (+241s) +ovms renderD130 0000:7c:00.0 +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 18640 after 0 ms: Could not connect to server +curl: (7) Failed to connect to 127.0.0.1 port 8090 after 0 ms: Could not connect to server +== load typical 00:19:31 (+257s) +== baseline typical 00:19:32 (+258s) +== solo hold 1 (AG), deadline +2250s 00:19:52 (+278s) +agentbench: AG: agent http://127.0.0.1:8090, proxy 127.0.0.1:18620 -> http://127.0.0.1:18640, pod cgroup /sys/fs/cgroup/system.slice/docker-0f3ecc5db6fc6a5011721ffe8eb1db36d348e0df866e428c31ae3e2c103f8c88.scope/kubepods/burstable/podaae2b796-aecd-4431-88ac-d979d4e06784 +agentbench: task 21: 1966 s left < 99999 s, hold ends +agentbench: solo hold 1 (AG): 0 complete attempt(s) +== finished 00:19:58 (+284s) +== cleanup 00:19:58 (+284s) +== done 00:20:00 (+286s) diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/orphans_killed b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/orphans_killed new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/orphans_killed @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/post.txt new file mode 100644 index 00000000..0c11eb8e --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/post.txt @@ -0,0 +1,15 @@ +2026-10-30T00:20:05Z + 00:20:05 up 44 days, 1:34, 10 users, load average: 8.61, 4.10, 3.05 +%CPU PID USER COMMAND +15.8 872705 65532 epp + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo + 0.5 2535 root containerd +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098586964 2538499400 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/pre.txt new file mode 100644 index 00000000..02e7dc9c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/pre.txt @@ -0,0 +1,15 @@ +2026-10-30T00:15:13Z + 00:15:13 up 44 days, 1:29, 10 users, load average: 5.09, 3.91, 2.91 +%CPU PID USER COMMAND +15.8 872705 65532 epp + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.6 1548406 root kworker/u384:1-events_power_efficient + 0.5 3882 10001 tempo +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098401352 2538685012 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/rc b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/rc new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/rc @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/serving.jsonl b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/serving.jsonl new file mode 100644 index 00000000..5c180b88 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/serving.jsonl @@ -0,0 +1,29 @@ +{"ts": 1793319324.14, "seconds": 10.0, "load1": 4.39, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.009, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.021, "xpumd": 0.002}, "serving_cpu": 0.095, "aiperf_procs": 0} +{"ts": 1793319334.168, "seconds": 10.028, "load1": 3.86, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.058, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.019, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.148, "aiperf_procs": 0} +{"ts": 1793319344.2, "seconds": 10.032, "load1": 3.27, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.008, "drm-exporter": 0.0, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319354.23, "seconds": 10.03, "load1": 3.08, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.008, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319364.258, "seconds": 10.028, "load1": 2.61, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319374.287, "seconds": 10.029, "load1": 2.21, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.007, "drm-exporter": 0.0, "loki": 0.022, "xpumd": 0.001}, "serving_cpu": 0.102, "aiperf_procs": 0} +{"ts": 1793319384.4, "seconds": 10.113, "load1": 1.87, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793319394.428, "seconds": 10.028, "load1": 1.58, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.08, "aiperf_procs": 0} +{"ts": 1793319404.457, "seconds": 10.028, "load1": 1.56, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.013, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.092, "aiperf_procs": 0} +{"ts": 1793319414.486, "seconds": 10.029, "load1": 1.32, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.011, "xpumd": 0.001}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793319424.518, "seconds": 10.032, "load1": 1.11, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.016, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.091, "aiperf_procs": 0} +{"ts": 1793319434.548, "seconds": 10.03, "load1": 0.94, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.022, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.098, "aiperf_procs": 0} +{"ts": 1793319444.643, "seconds": 10.095, "load1": 0.8, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.082, "aiperf_procs": 0} +{"ts": 1793319454.671, "seconds": 10.028, "load1": 0.67, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.006, "xpumd": 0.001}, "serving_cpu": 0.08, "aiperf_procs": 0} +{"ts": 1793319464.701, "seconds": 10.03, "load1": 0.57, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.001}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319474.734, "seconds": 10.033, "load1": 0.48, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793319484.766, "seconds": 10.032, "load1": 0.41, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.008, "otel-collector": 0.0, "tempo": 0.004, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.006, "xpumd": 0.003}, "serving_cpu": 0.08, "aiperf_procs": 0} +{"ts": 1793319494.796, "seconds": 10.03, "load1": 0.34, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.016, "xpumd": 0.002}, "serving_cpu": 0.098, "aiperf_procs": 0} +{"ts": 1793319504.918, "seconds": 10.122, "load1": 0.45, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793319514.946, "seconds": 10.028, "load1": 0.38, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.014, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793319524.976, "seconds": 10.03, "load1": 0.64, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.014, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.012, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.003}, "serving_cpu": 0.101, "aiperf_procs": 0} +{"ts": 1793319535.006, "seconds": 10.03, "load1": 0.69, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.091, "aiperf_procs": 0} +{"ts": 1793319545.038, "seconds": 10.032, "load1": 0.74, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.022, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.117, "aiperf_procs": 0} +{"ts": 1793319555.068, "seconds": 10.03, "load1": 1.01, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.013, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.027, "prometheus": 0.007, "alloy": 0.008, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.096, "aiperf_procs": 0} +{"ts": 1793319565.176, "seconds": 10.108, "load1": 1.01, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793319575.208, "seconds": 10.032, "load1": 7.01, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.036, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.027, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.116, "aiperf_procs": 0} +{"ts": 1793319585.244, "seconds": 10.036, "load1": 8.53, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.019, "grafana": 0.008, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.03, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.013, "xpumd": 0.002}, "serving_cpu": 0.11, "aiperf_procs": 0} +{"ts": 1793319595.278, "seconds": 10.034, "load1": 9.99, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.014, "grafana": 0.006, "otel-collector": 0.004, "tempo": 0.006, "node-exporter": 0.028, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.004}, "serving_cpu": 0.103, "aiperf_procs": 0} +{"ts": 1793319605.309, "seconds": 10.031, "load1": 8.61, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.009, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.029, "prometheus": 0.007, "alloy": 0.009, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.095, "aiperf_procs": 0} diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/stale_reaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/stale_reaped new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/stale_reaped @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/unreaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/unreaped new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/wait_s b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-AG/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-post.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-post.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-pre.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/dmesg-pre.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/hold.log b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/hold.log new file mode 100644 index 00000000..aba49025 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/hold.log @@ -0,0 +1,7 @@ +== arm H53 start 00:20:14 (+0s) +== k3d-gpu up 00:20:15 (+1s) +== wait-ready 00:23:06 (+172s) +Waiting for deployment "vllm-qwen38-27b-xpu" rollout to finish: 0 of 1 updated replicas are available... +== cleanup 00:25:19 (+305s) +== done 00:25:22 (+308s) +Terminated "$@" diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/orphans_killed b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/orphans_killed new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/orphans_killed @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/post.txt new file mode 100644 index 00000000..19586570 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/post.txt @@ -0,0 +1,15 @@ +2026-10-30T00:25:26Z + 00:25:26 up 44 days, 1:39, 11 users, load average: 0.80, 1.80, 2.32 +%CPU PID USER COMMAND +15.7 872705 65532 epp + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo + 0.5 2535 root containerd +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098591344 2538495020 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/pre.txt new file mode 100644 index 00000000..2da69f6c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/pre.txt @@ -0,0 +1,15 @@ +2026-10-30T00:20:14Z + 00:20:14 up 44 days, 1:34, 10 users, load average: 7.92, 4.03, 3.03 +%CPU PID USER COMMAND +15.8 872705 65532 epp + 4.0 1565186 devcloud bash + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098583812 2538502552 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/rc b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/rc new file mode 100644 index 00000000..fba7ed52 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/rc @@ -0,0 +1 @@ +143 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/serving.jsonl b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/serving.jsonl new file mode 100644 index 00000000..36ca245f --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/serving.jsonl @@ -0,0 +1,31 @@ +{"ts": 1793319624.968, "seconds": 10.0, "load1": 6.77, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.004}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793319634.997, "seconds": 10.029, "load1": 5.8, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.039, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.001}, "serving_cpu": 0.118, "aiperf_procs": 0} +{"ts": 1793319645.028, "seconds": 10.03, "load1": 4.91, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.008, "otel-collector": 0.001, "tempo": 0.022, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.014, "drm-exporter": 0.0, "loki": 0.013, "xpumd": 0.003}, "serving_cpu": 0.114, "aiperf_procs": 0} +{"ts": 1793319655.056, "seconds": 10.029, "load1": 4.8, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.008, "drm-exporter": 0.0, "loki": 0.024, "xpumd": 0.002}, "serving_cpu": 0.101, "aiperf_procs": 0} +{"ts": 1793319665.087, "seconds": 10.031, "load1": 3.8, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.017, "otel-collector": 0.0, "tempo": 0.004, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.095, "aiperf_procs": 0} +{"ts": 1793319675.122, "seconds": 10.034, "load1": 3.3, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793319685.231, "seconds": 10.109, "load1": 2.79, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.083, "aiperf_procs": 0} +{"ts": 1793319695.261, "seconds": 10.03, "load1": 2.36, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.079, "aiperf_procs": 0} +{"ts": 1793319705.293, "seconds": 10.033, "load1": 2.0, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.082, "aiperf_procs": 0} +{"ts": 1793319715.325, "seconds": 10.032, "load1": 1.69, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.001}, "serving_cpu": 0.091, "aiperf_procs": 0} +{"ts": 1793319725.354, "seconds": 10.029, "load1": 1.43, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.009, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793319735.387, "seconds": 10.033, "load1": 1.21, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.009, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793319745.498, "seconds": 10.111, "load1": 1.02, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319755.528, "seconds": 10.03, "load1": 0.86, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.021, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.096, "aiperf_procs": 0} +{"ts": 1793319765.555, "seconds": 10.027, "load1": 0.96, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.01, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319775.592, "seconds": 10.037, "load1": 0.81, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.016, "xpumd": 0.002}, "serving_cpu": 0.093, "aiperf_procs": 0} +{"ts": 1793319785.622, "seconds": 10.03, "load1": 0.69, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.017, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.01, "xpumd": 0.003}, "serving_cpu": 0.097, "aiperf_procs": 0} +{"ts": 1793319795.651, "seconds": 10.029, "load1": 0.65, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.085, "aiperf_procs": 0} +{"ts": 1793319805.754, "seconds": 10.103, "load1": 0.87, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.082, "aiperf_procs": 0} +{"ts": 1793319815.785, "seconds": 10.031, "load1": 0.74, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.006, "xpumd": 0.003}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793319825.816, "seconds": 10.031, "load1": 0.71, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793319835.848, "seconds": 10.032, "load1": 0.6, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793319845.88, "seconds": 10.032, "load1": 0.66, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793319855.915, "seconds": 10.035, "load1": 0.63, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.017, "node-exporter": 0.024, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.103, "aiperf_procs": 0} +{"ts": 1793319866.026, "seconds": 10.112, "load1": 0.61, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793319876.058, "seconds": 10.031, "load1": 0.82, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793319886.092, "seconds": 10.034, "load1": 0.93, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.016, "xpumd": 0.003}, "serving_cpu": 0.1, "aiperf_procs": 0} +{"ts": 1793319896.125, "seconds": 10.033, "load1": 0.94, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793319906.155, "seconds": 10.03, "load1": 0.79, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.017, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.011, "xpumd": 0.003}, "serving_cpu": 0.095, "aiperf_procs": 0} +{"ts": 1793319916.188, "seconds": 10.033, "load1": 0.67, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793319926.29, "seconds": 10.102, "load1": 0.8, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.009, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.021, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.081, "aiperf_procs": 0} diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/stale_reaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/stale_reaped new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/stale_reaped @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/unreaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/unreaped new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/wait_s b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-a/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-post.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-post.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-pre.txt new file mode 100644 index 00000000..1816dc93 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/dmesg-pre.txt @@ -0,0 +1,200 @@ +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:48:00.0: [drm] Check your /sys/class/drm/card2/device/devcoredump/data +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Xe device coredump has been created +[Sun Sep 27 03:57:29 2026] xe 0000:12:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data +[Sun Sep 27 04:38:15 2026] xe 0000:96:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 04:38:15 2026] xe 0000:7c:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:12:00.0: [drm] Xe device coredump has been deleted. +[Sun Sep 27 05:00:05 2026] xe 0000:48:00.0: [drm] Xe device coredump has been deleted. diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/hold.log b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/hold.log new file mode 100644 index 00000000..72e77adc --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/hold.log @@ -0,0 +1,7 @@ +== arm H53 start 00:25:35 (+0s) +== k3d-gpu up 00:25:35 (+0s) +== wait-ready 00:28:27 (+172s) +Waiting for deployment "vllm-qwen38-27b-xpu" rollout to finish: 0 of 1 updated replicas are available... +== cleanup 00:36:21 (+646s) +== done 00:36:23 (+648s) +Terminated "$@" diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/orphans_killed b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/orphans_killed new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/orphans_killed @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/post.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/post.txt new file mode 100644 index 00000000..74fb4786 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/post.txt @@ -0,0 +1,15 @@ +2026-10-30T00:36:28Z + 00:36:28 up 44 days, 1:50, 11 users, load average: 5.38, 2.98, 2.37 +%CPU PID USER COMMAND +15.7 872705 65532 epp + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo + 0.5 2535 root containerd +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098604252 2538482112 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/pre.txt b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/pre.txt new file mode 100644 index 00000000..e2c3c3da --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/pre.txt @@ -0,0 +1,15 @@ +2026-10-30T00:25:35Z + 00:25:35 up 44 days, 1:40, 10 users, load average: 0.91, 1.79, 2.31 +%CPU PID USER COMMAND +15.7 872705 65532 epp + 3.8 1578915 devcloud bash + 2.6 3911 nobody node_exporter + 1.2 3934 root python3 + 1.0 2651 root dockerd + 1.0 3892 10001 loki + 0.6 3941 472 grafana + 0.6 3936 nobody prometheus + 0.6 3914 root alloy + 0.5 3882 10001 tempo +aiperf_procs: 0 +/dev/nvme0n1p1 3637102748 1098591488 2538494876 31% / diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/rc b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/rc new file mode 100644 index 00000000..fba7ed52 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/rc @@ -0,0 +1 @@ +143 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/serving.jsonl b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/serving.jsonl new file mode 100644 index 00000000..feb6fcbf --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/serving.jsonl @@ -0,0 +1,65 @@ +{"ts": 1793319945.496, "seconds": 10.0, "load1": 1.08, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.012, "aiperf010": 0.0, "xe-exporter": 0.055, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.018, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.15, "aiperf_procs": 0} +{"ts": 1793319955.527, "seconds": 10.031, "load1": 1.06, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.002}, "serving_cpu": 0.085, "aiperf_procs": 0} +{"ts": 1793319965.557, "seconds": 10.03, "load1": 1.05, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.009, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.008, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.083, "aiperf_procs": 0} +{"ts": 1793319975.587, "seconds": 10.03, "load1": 0.96, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.008, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793319985.616, "seconds": 10.029, "load1": 0.82, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793319995.654, "seconds": 10.038, "load1": 0.76, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.006, "xpumd": 0.002}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793320005.756, "seconds": 10.102, "load1": 0.65, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.011, "drm-exporter": 0.0, "loki": 0.029, "xpumd": 0.001}, "serving_cpu": 0.111, "aiperf_procs": 0} +{"ts": 1793320015.785, "seconds": 10.029, "load1": 0.55, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793320025.818, "seconds": 10.032, "load1": 0.46, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.017, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320035.847, "seconds": 10.029, "load1": 0.39, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793320045.878, "seconds": 10.031, "load1": 0.33, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.008, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.078, "aiperf_procs": 0} +{"ts": 1793320055.906, "seconds": 10.028, "load1": 0.28, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793320066.005, "seconds": 10.098, "load1": 0.24, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.012, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793320076.034, "seconds": 10.03, "load1": 0.36, "cpu": {"dynamo-frontend": 0.004, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.011, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320086.062, "seconds": 10.028, "load1": 0.38, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.0, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.079, "aiperf_procs": 0} +{"ts": 1793320096.103, "seconds": 10.041, "load1": 0.32, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320106.134, "seconds": 10.031, "load1": 0.27, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.001}, "serving_cpu": 0.078, "aiperf_procs": 0} +{"ts": 1793320116.162, "seconds": 10.028, "load1": 0.54, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793320126.281, "seconds": 10.119, "load1": 0.46, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.021, "xpumd": 0.003}, "serving_cpu": 0.105, "aiperf_procs": 0} +{"ts": 1793320136.311, "seconds": 10.03, "load1": 0.47, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.083, "aiperf_procs": 0} +{"ts": 1793320146.342, "seconds": 10.031, "load1": 0.78, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.021, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.112, "aiperf_procs": 0} +{"ts": 1793320156.373, "seconds": 10.031, "load1": 0.89, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320166.407, "seconds": 10.034, "load1": 0.98, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.085, "aiperf_procs": 0} +{"ts": 1793320176.438, "seconds": 10.031, "load1": 1.13, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.085, "aiperf_procs": 0} +{"ts": 1793320186.539, "seconds": 10.101, "load1": 1.19, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.003}, "serving_cpu": 0.092, "aiperf_procs": 0} +{"ts": 1793320196.573, "seconds": 10.035, "load1": 1.08, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.083, "aiperf_procs": 0} +{"ts": 1793320206.605, "seconds": 10.032, "load1": 1.21, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320216.639, "seconds": 10.034, "load1": 1.25, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.091, "aiperf_procs": 0} +{"ts": 1793320226.673, "seconds": 10.034, "load1": 1.21, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320236.708, "seconds": 10.034, "load1": 1.18, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.024, "xpumd": 0.003}, "serving_cpu": 0.103, "aiperf_procs": 0} +{"ts": 1793320246.806, "seconds": 10.098, "load1": 1.3, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.008, "otel-collector": 0.001, "tempo": 0.024, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.002}, "serving_cpu": 0.115, "aiperf_procs": 0} +{"ts": 1793320256.838, "seconds": 10.032, "load1": 1.34, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320266.87, "seconds": 10.032, "load1": 1.29, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.097, "aiperf_procs": 0} +{"ts": 1793320276.902, "seconds": 10.031, "load1": 1.46, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793320286.931, "seconds": 10.029, "load1": 1.7, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320296.961, "seconds": 10.03, "load1": 1.59, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320307.07, "seconds": 10.109, "load1": 1.5, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.003}, "serving_cpu": 0.094, "aiperf_procs": 0} +{"ts": 1793320317.102, "seconds": 10.032, "load1": 1.42, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.013, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320327.133, "seconds": 10.031, "load1": 1.36, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.003}, "serving_cpu": 0.092, "aiperf_procs": 0} +{"ts": 1793320337.165, "seconds": 10.031, "load1": 1.3, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320347.196, "seconds": 10.031, "load1": 1.33, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.008, "xpumd": 0.002}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320357.226, "seconds": 10.03, "load1": 1.28, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.003}, "serving_cpu": 0.096, "aiperf_procs": 0} +{"ts": 1793320367.331, "seconds": 10.105, "load1": 1.24, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.012, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.011, "xpumd": 0.002}, "serving_cpu": 0.096, "aiperf_procs": 0} +{"ts": 1793320377.366, "seconds": 10.035, "load1": 1.2, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.087, "aiperf_procs": 0} +{"ts": 1793320387.399, "seconds": 10.034, "load1": 1.17, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.101, "aiperf_procs": 0} +{"ts": 1793320397.433, "seconds": 10.033, "load1": 1.22, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320407.461, "seconds": 10.029, "load1": 2.88, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320417.495, "seconds": 10.033, "load1": 2.59, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.013, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.026, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320427.598, "seconds": 10.104, "load1": 2.48, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.003}, "serving_cpu": 0.095, "aiperf_procs": 0} +{"ts": 1793320437.632, "seconds": 10.034, "load1": 2.1, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.081, "aiperf_procs": 0} +{"ts": 1793320447.662, "seconds": 10.03, "load1": 6.98, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320457.692, "seconds": 10.03, "load1": 6.06, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.02, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.103, "aiperf_procs": 0} +{"ts": 1793320467.724, "seconds": 10.032, "load1": 5.28, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320477.753, "seconds": 10.029, "load1": 4.62, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.011, "drm-exporter": 0.001, "loki": 0.016, "xpumd": 0.002}, "serving_cpu": 0.101, "aiperf_procs": 0} +{"ts": 1793320487.862, "seconds": 10.109, "load1": 4.22, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.004, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.012, "xpumd": 0.003}, "serving_cpu": 0.09, "aiperf_procs": 0} +{"ts": 1793320497.893, "seconds": 10.031, "load1": 3.73, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.012, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.089, "aiperf_procs": 0} +{"ts": 1793320507.924, "seconds": 10.031, "load1": 3.38, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.017, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.1, "aiperf_procs": 0} +{"ts": 1793320517.953, "seconds": 10.029, "load1": 3.17, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.007, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320527.985, "seconds": 10.032, "load1": 2.84, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.012, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.006, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.088, "aiperf_procs": 0} +{"ts": 1793320538.016, "seconds": 10.031, "load1": 2.47, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.025, "prometheus": 0.007, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320548.11, "seconds": 10.094, "load1": 2.17, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.011, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.007, "otel-collector": 0.003, "tempo": 0.005, "node-exporter": 0.022, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.086, "aiperf_procs": 0} +{"ts": 1793320558.141, "seconds": 10.031, "load1": 1.84, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.005, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.0, "loki": 0.011, "xpumd": 0.001}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793320568.173, "seconds": 10.032, "load1": 1.63, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.01, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.009, "otel-collector": 0.001, "tempo": 0.019, "node-exporter": 0.024, "prometheus": 0.01, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.106, "aiperf_procs": 0} +{"ts": 1793320578.204, "seconds": 10.03, "load1": 6.18, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.011, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.024, "prometheus": 0.006, "alloy": 0.006, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.002}, "serving_cpu": 0.084, "aiperf_procs": 0} +{"ts": 1793320588.237, "seconds": 10.033, "load1": 5.38, "cpu": {"dynamo-frontend": 0.003, "llmd-envoy": 0.003, "llmd-epp": 0.009, "aiperf010": 0.0, "xe-exporter": 0.01, "grafana": 0.006, "otel-collector": 0.001, "tempo": 0.005, "node-exporter": 0.023, "prometheus": 0.007, "alloy": 0.007, "drm-exporter": 0.001, "loki": 0.007, "xpumd": 0.003}, "serving_cpu": 0.085, "aiperf_procs": 0} diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/stale_reaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/stale_reaped new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/stale_reaped @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/unreaped b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/unreaped new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/wait_s b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/runs/H53-smoke-B2-b/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.err b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.err new file mode 100644 index 00000000..8ba33fa5 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.err @@ -0,0 +1,2 @@ +==> fit: OpenVINO/Qwen3.8-27B-int4-ov on 1 GPU via ovms, context 32768 x 6 seqs, GPU 28.5 GiB/card, host 16.6 GiB (pod limit 19Gi), host reserve 50.3 GiB +==> agent: OpenVINO/Qwen3.8-27B-int4-ov as 'qwen3.8-27b' on GPU (gpu.intel.com/xe), 2 CPU(s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.log new file mode 100644 index 00000000..75d2c8d6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.log @@ -0,0 +1,48 @@ +Langfuse not configured: set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY to enable tracing +Resolved MCP client class: agent_framework.MCPStreamableHTTPTool +Banking MCP client configured for http://127.0.0.1:18600/mcp +============================================================ +Banking Assistant with DevUI +============================================================ +Starting web interface at http://localhost:8090 + +Features: + • Natural-language banking queries + • Banking tools via MCP server: http://127.0.0.1:18600/mcp + • Persistent memory via Redis (text-only) + • OpenAI-compatible API (GenAI Gateway / LiteLLM) + +Prerequisites: + 1. Banking MCP server reachable at MCP_SERVER_URL + 2. Redis: docker run -d -p 6379:6379 redis/redis-stack-server + 3. Configure .env with your LLM endpoint + +============================================================ +Registering 1 in-memory entities +Authentication middleware enabled +Starting Agent Framework DevUI on 127.0.0.1:8090 +INFO: Started server process [1563224] +INFO: Waiting for application startup. +Starting Agent Framework Server +Initializing Agent Framework executor... +Set up TracerProvider for instrumentation +Enabled Agent Framework observability +No Agent Framework entities directory configured +Discovered 0 entities from directory +Registered in-memory entity: agent_in_memory_banking-assistant_359c4828784a45298ac97575b50b5a42 +Total entities available: 1 +OpenAI proxy mode available (OPENAI_API_KEY configured) +INFO: Application startup complete. +INFO: Uvicorn running on http://127.0.0.1:8090 (Press CTRL+C to quit) +INFO: 127.0.0.1:54408 - "GET /v1/entities HTTP/1.1" 200 OK +INFO: 127.0.0.1:55396 - "GET /v1/entities HTTP/1.1" 200 OK +Extracted entity_id: agent_in_memory_banking-assistant_359c4828784a45298ac97575b50b5a42 +Found entity: banking-assistant (agent) +Executing agent: agent_in_memory_banking-assistant_359c4828784a45298ac97575b50b5a42 +Negotiated protocol version: 2025-11-25 +HTTP Request: POST http://127.0.0.1:18620/v1/chat/completions "HTTP/1.1 200 OK" +Function name: get_accounts +Function get_accounts succeeded. +Function duration: 0.004757s +HTTP Request: POST http://127.0.0.1:18620/v1/chat/completions "HTTP/1.1 200 OK" +INFO: 127.0.0.1:55398 - "POST /v1/responses HTTP/1.1" 200 OK diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.sh b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.sh new file mode 100755 index 00000000..01e617f9 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# (Re)start the banking agent and wait for its DevUI. +[ -f '/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.pid' ] && kill "$(cat '/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.pid')" 2>/dev/null +for _ in $(seq 100); do (exec 3<>/dev/tcp/127.0.0.1/8090) 2>/dev/null || break; sleep 0.1; done +cd '/home/devcloud/k3dgpu-bench/eab/usecases/banking-assistant/src' && (OPENAI_BASE_URL=http://127.0.0.1:18620/v1 OPENAI_API_KEY=not-needed MODEL_NAME='Qwen/Qwen3.8-27B' \ + MCP_SERVER_URL=http://127.0.0.1:18600/mcp ENABLE_MEMORY=false AUTO_OPEN_BROWSER=false \ + DEVUI_HOST=127.0.0.1 exec '/home/devcloud/k3dgpu-bench/mafvenv/bin/python' main.py >> '/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.log' 2>&1 & + echo $! > '/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.pid') +for _ in $(seq 45); do + curl -fsS -o /dev/null --max-time 5 -H "Authorization: Bearer ${DEVUI_AUTH_TOKEN}" http://127.0.0.1:8090/v1/entities && exit 0 + sleep 2 +done +exit 1 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.yaml new file mode 100644 index 00000000..1faad8cc --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/agent.yaml @@ -0,0 +1,112 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: k3d-gpu-agent +--- +# Models and the compiled-model cache live here, so restarts never re-download. +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: ovms-models + namespace: k3d-gpu-agent +spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 20Gi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ovms + namespace: k3d-gpu-agent + labels: + app: ovms +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: ovms + template: + metadata: + labels: + app: ovms + annotations: + k3d-gpu.io/agent-device: GPU + k3d-gpu.io/agent-fit: "gpu 6x32768" + spec: + securityContext: + runAsUser: 5000 + runAsGroup: 5000 + fsGroup: 5000 + supplementalGroups: [990] + nodeSelector: + gpu.intel.com/xmx: "true" + initContainers: + # --pull downloads the model once and writes its graph.pbtxt (device, + # parsers). On later starts the files are already there and only the + # graph is rewritten, so a device change takes effect with no download. + - name: pull + image: openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --pull + - --source_model=OpenVINO/Qwen3.8-27B-int4-ov + - --model_repository_path=/models + - --model_name=qwen3.8-27b + - --task=text_generation + - --target_device=GPU + - --tool_parser=qwen3coder + - --max_num_seqs=6 + - --cache_size=12 + volumeMounts: + - name: models + mountPath: /models + containers: + - name: ovms + image: openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --rest_port=8000 + - --model_path=/models/OpenVINO/Qwen3.8-27B-int4-ov + - --model_name=qwen3.8-27b + - --cache_dir=/models/.ov_cache + - --log_level=INFO + ports: + - name: http + containerPort: 8000 + # No liveness probe: compiling a model for a GPU takes minutes. + readinessProbe: + httpGet: + path: /v2/health/ready + port: http + periodSeconds: 5 + resources: + requests: + cpu: "2" + memory: 19Gi + limits: + memory: 19Gi + gpu.intel.com/xe: 1 + volumeMounts: + - name: models + mountPath: /models + volumes: + - name: models + persistentVolumeClaim: + claimName: ovms-models +--- +apiVersion: v1 +kind: Service +metadata: + name: ovms + namespace: k3d-gpu-agent +spec: + selector: + app: ovms + ports: + - name: http + port: 8000 + targetPort: http diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/b-listing.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/b-listing.txt new file mode 100644 index 00000000..de902c4d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/b-listing.txt @@ -0,0 +1,34 @@ +.gitattributes 1570 1793310278.7959648510 +LICENSE 11544 1793310278.8089651410 +README.md 65012 1793310278.8199653860 +chat_template.jinja 8952 1793310278.8139652520 +config.json 4312 1793310278.8129652300 +crc32.txt 238 1793310278.8299656090 +generation_config.json 202 1793310278.8259655200 +merges.txt 3353259 1793310278.8589662550 +model-00001-of-00018.safetensors 3966730552 1793310507.8016561670 +model-00002-of-00018.safetensors 3043080328 1793310511.8507358520 +model-00003-of-00018.safetensors 2542796952 1793310459.7537105830 +model-00004-of-00018.safetensors 3988973152 1793310571.2509048500 +model-00005-of-00018.safetensors 2099339864 1793310430.8621419970 +model-00006-of-00018.safetensors 3979553696 1793310509.6426923980 +model-00007-of-00018.safetensors 2108759344 1793310436.7122571270 +model-00008-of-00018.safetensors 3979553696 1793310573.9919587940 +model-00009-of-00018.safetensors 2108759344 1793310591.7193076680 +model-00010-of-00018.safetensors 3979553696 1793310721.1158541950 +model-00011-of-00018.safetensors 2108759344 1793310632.3041063770 +model-00012-of-00018.safetensors 3979553696 1793310730.4580380490 +model-00013-of-00018.safetensors 2108759344 1793310660.3256578430 +model-00014-of-00018.safetensors 3979553696 1793310752.6844754650 +model-00015-of-00018.safetensors 2108759344 1793310664.6107421730 +model-00016-of-00018.safetensors 3979564040 1793310758.3635872300 +model-00017-of-00018.safetensors 2108759344 1793310689.8802394770 +model-00018-of-00018.safetensors 3392197344 1793310751.2064463780 +model.safetensors.index.json 112216 1793310660.4686606570 +preprocessor_config.json 390 1793310660.5806628610 +tokenizer.json 12809320 1793310661.7056850020 +tokenizer_config.json 17928 1793310661.8266873830 +video_preprocessor_config.json 385 1793310661.9656901190 +vocab.json 6722759 1793310662.3626979320 +77042094076611b69791a610065f28b7013b8c621795fa86ddccc8bac7d1b9df model.safetensors.index.json +191e0af232104ed8b65258cf3fb2b842e288008baca7633c11b82a1ac7203aab config.json diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/blueprint-commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/blueprint-commit.txt new file mode 100644 index 00000000..2cf3b989 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/blueprint-commit.txt @@ -0,0 +1 @@ +758d4845520ca5456bf732264ef02d92a26d25a6 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/commit.txt new file mode 100644 index 00000000..4af4a96a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/commit.txt @@ -0,0 +1 @@ +1f39488be7695dc0d1848347bf1c91f04025bc55 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/deploy.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/deploy.yaml new file mode 100644 index 00000000..33752ef7 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/deploy.yaml @@ -0,0 +1,98 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: k3d-gpu-agent +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ovms + namespace: k3d-gpu-agent + labels: + app: ovms +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: ovms + template: + metadata: + labels: + app: ovms + annotations: + k3d-gpu.io/agent-device: GPU + k3d-gpu.io/agent-fit: gpu 6x32768 + spec: + securityContext: + runAsUser: 5000 + runAsGroup: 5000 + fsGroup: 5000 + supplementalGroups: + - 990 + nodeSelector: + gpu.intel.com/xmx: 'true' + initContainers: + - name: pull + image: openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --pull + - --source_model=OpenVINO/Qwen3.8-27B-int4-ov + - --model_repository_path=/models + - --model_name=qwen3.8-27b + - --task=text_generation + - --target_device=GPU + - --tool_parser=qwen3coder + - --max_num_seqs=6 + - --cache_size=12 + volumeMounts: + - name: models + mountPath: /models + containers: + - name: ovms + image: openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --rest_port=8000 + - --model_path=/models/OpenVINO/Qwen3.8-27B-int4-ov + - --model_name=qwen3.8-27b + - --cache_dir=/models/.ov_cache + - --log_level=INFO + ports: + - name: http + containerPort: 8000 + readinessProbe: + httpGet: + path: /v2/health/ready + port: http + periodSeconds: 5 + resources: + requests: + cpu: '2' + memory: 19Gi + limits: + memory: 19Gi + gpu.intel.com/xe: 1 + volumeMounts: + - name: models + mountPath: /models + volumes: + - name: models + hostPath: + path: /var/apb-models + type: Directory +--- +apiVersion: v1 +kind: Service +metadata: + name: ovms + namespace: k3d-gpu-agent +spec: + selector: + app: ovms + ports: + - name: http + port: 8000 + targetPort: http diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-arms.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-arms.txt new file mode 100644 index 00000000..f44d9837 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-arms.txt @@ -0,0 +1 @@ +ovms renderD130 0000:7c:00.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-hold.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-hold.txt new file mode 100644 index 00000000..971e9a12 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/gpu-hold.txt @@ -0,0 +1,4 @@ +gpu-hold-1 renderD128 0000:12:00.0 +gpu-hold-2 renderD130 0000:7c:00.0 +gpu-hold-3 renderD129 0000:48:00.0 +gpu-hold-4 renderD131 0000:96:00.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/image.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/image.txt new file mode 100644 index 00000000..3fa78339 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/image.txt @@ -0,0 +1 @@ +aae2b796-aecd-4431-88ac-d979d4e06784 openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 docker.io/openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 \ No newline at end of file diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm-image.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm-image.txt new file mode 100644 index 00000000..89987653 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm-image.txt @@ -0,0 +1 @@ +sha256:73cbb215d3d43ae5b42816217b200386eb08d567767fcb36cba163152c5f3c79 docker.litellm.ai/berriai/litellm:v1.83.14-stable.patch.2@sha256:73cbb215d3d43ae5b42816217b200386eb08d567767fcb36cba163152c5f3c79 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm.yaml new file mode 100644 index 00000000..78b1e4d6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/litellm.yaml @@ -0,0 +1,7 @@ +model_list: + - model_name: Qwen/Qwen3.8-27B + litellm_params: + model: openai/qwen3.8-27b + api_base: http://127.0.0.1:18630/v1 + api_key: not-needed + timeout: 600 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/live.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/live.log new file mode 100644 index 00000000..4d34134d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/live.log @@ -0,0 +1,13 @@ +== arm H53 start 00:15:14 (+0s) +== k3d-gpu up 00:15:16 (+2s) +== wait-ready 00:18:24 (+190s) +== ready 00:19:15 (+241s) +== load typical 00:19:31 (+257s) +== baseline typical 00:19:32 (+258s) +== solo hold 1 (AG), deadline +2250s 00:19:52 (+278s) +agentbench: AG: agent http://127.0.0.1:8090, proxy 127.0.0.1:18620 -> http://127.0.0.1:18640, pod cgroup /sys/fs/cgroup/system.slice/docker-0f3ecc5db6fc6a5011721ffe8eb1db36d348e0df866e428c31ae3e2c103f8c88.scope/kubepods/burstable/podaae2b796-aecd-4431-88ac-d979d4e06784 +agentbench: task 21: 1966 s left < 99999 s, hold ends +agentbench: solo hold 1 (AG): 0 complete attempt(s) +== finished 00:19:58 (+284s) +== cleanup 00:19:58 (+284s) +== done 00:20:00 (+286s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/load-start.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/load-start.log new file mode 100644 index 00000000..88743ad2 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/load-start.log @@ -0,0 +1 @@ +started typical on 24-47,72-95 (3 containers, ttl 2173s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/mcp.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/mcp.log new file mode 100644 index 00000000..aa260a00 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/mcp.log @@ -0,0 +1,30 @@ +Starting Banking MCP server on 127.0.0.1:18600 (path /mcp) +INFO: Started server process [1563213] +INFO: Waiting for application startup. +StreamableHTTP session manager started +INFO: Application startup complete. +INFO: Uvicorn running on http://127.0.0.1:18600 (Press CTRL+C to quit) +INFO: 127.0.0.1:45244 - "POST /mcp HTTP/1.1" 200 OK +Terminating session: None +INFO: 127.0.0.1:45256 - "POST /mcp HTTP/1.1" 202 Accepted +Terminating session: None +INFO: 127.0.0.1:45264 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type PingRequest +Terminating session: None +INFO: 127.0.0.1:45278 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type ListToolsRequest +Terminating session: None +INFO: 127.0.0.1:45288 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type PingRequest +Terminating session: None +INFO: 127.0.0.1:45302 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type ListPromptsRequest +Terminating session: None +INFO: 127.0.0.1:44382 - "POST /mcp HTTP/1.1" 200 OK +Processing request of type CallToolRequest +Terminating session: None +INFO: Shutting down +INFO: Waiting for application shutdown. +StreamableHTTP session manager shutting down +INFO: Application shutdown complete. +INFO: Finished server process [1563213] diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/models.json b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/models.json new file mode 100644 index 00000000..f5aae991 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/models.json @@ -0,0 +1 @@ +{"data":[{"id":"qwen3.8-27b","object":"model","created":1793319555,"owned_by":"OVMS"}],"object":"list"} \ No newline at end of file diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/pf.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/pf.log new file mode 100644 index 00000000..40e3e1b8 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/pf.log @@ -0,0 +1,5 @@ +Forwarding from 127.0.0.1:18630 -> 8000 +Forwarding from [::1]:18630 -> 8000 +Handling connection for 18630 +Handling connection for 18630 +Handling connection for 18630 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/reset-mcp.sh b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/reset-mcp.sh new file mode 100755 index 00000000..01314a4c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/reset-mcp.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +kill "$(cat "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/mcp.pid")" 2>/dev/null +for _ in $(seq 50); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null || break; sleep 0.1; done +cd "/home/devcloud/k3dgpu-bench/eab/usecases/banking-assistant/src" && (MCP_HOST=127.0.0.1 MCP_PORT=18600 exec "/home/devcloud/k3dgpu-bench/mafvenv/bin/python" -m mcp_server.server >> "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/mcp.log" 2>&1 & + echo $! > "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/mcp.pid") +for _ in $(seq 120); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null && exit 0; sleep 0.5; done +exit 1 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/up.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/up.log new file mode 100644 index 00000000..ae57c60d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/hold/up.log @@ -0,0 +1,33 @@ +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators: intel amx +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-e-h53' with image cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-e-h53' +INFO[0000] Created image volume k3d-k3dgpu-e-h53-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-e-h53-tools' +INFO[0001] Creating node 'k3d-k3dgpu-e-h53-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-e-h53' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-e-h53-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-e-h53' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-e-h53) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/kubeconfig +==> Waiting for nodes Ready... +node/k3d-k3dgpu-e-h53-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +==> intel device plugin auto-deployed by k3s. +==> Verifying accelerators are schedulable... +Waiting for daemon set "intel-gpu-plugin" rollout to finish: 0 of 1 updated pods are available... +daemon set "intel-gpu-plugin" successfully rolled out +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators schedulable: intel=4 amx=1. Try: k3d-gpu test diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/baseline-hold1.jsonl b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/baseline-hold1.jsonl new file mode 100644 index 00000000..0acd80b8 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/baseline-hold1.jsonl @@ -0,0 +1,2 @@ +{"ts": 1793319582.2942176, "tag": "baseline", "level": "typical", "seconds": 10.055, "cpus": "24-47,72-95", "busy_pct": 36.87, "load1": 7.97, "pkg_w": 188.04, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "baseline", "busy_cpus": "24-47,72-95", "load_expected": 3, "load_running": 3} +{"ts": 1793319592.4581883, "tag": "baseline", "level": "typical", "seconds": 10.056, "cpus": "24-47,72-95", "busy_pct": 36.84, "load1": 9.29, "pkg_w": 188.62, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "baseline", "busy_cpus": "24-47,72-95", "load_expected": 3, "load_running": 3} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/facts-hold1.json b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/facts-hold1.json new file mode 100644 index 00000000..faa13e7b --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/facts-hold1.json @@ -0,0 +1,349 @@ +{ + "argv": [ + "/home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/agentbench.py", + "solo", + "--hyp", + "H53", + "--hold", + "1", + "--arm", + "AG", + "--out", + "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs", + "--deadline", + "1793321564", + "--tasks", + "B24", + "--agent", + "http://127.0.0.1:8090", + "--upstream", + "http://127.0.0.1:18640", + "--proxy-port", + "18620", + "--restart", + "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.sh", + "--pod", + "/sys/fs/cgroup/system.slice/docker-0f3ecc5db6fc6a5011721ffe8eb1db36d348e0df866e428c31ae3e2c103f8c88.scope/kubepods/burstable/podaae2b796-aecd-4431-88ac-d979d4e06784", + "--mcp-url", + "http://127.0.0.1:18600/mcp", + "--reset", + "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/reset-mcp.sh", + "--sampling", + "agent_qwen38", + "--baseline", + "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs/baseline-hold1.jsonl", + "--level", + "typical", + "--guard-s", + "99999" + ], + "args": { + "cmd": "solo", + "hyp": "H53", + "hold": 1, + "arm": "AG", + "out": "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs", + "deadline": 1793321564.0, + "tasks": "B24", + "agent": "http://127.0.0.1:8090", + "upstream": "http://127.0.0.1:18640", + "proxy_port": 18620, + "restart": "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/agent.sh", + "pod": "/sys/fs/cgroup/system.slice/docker-0f3ecc5db6fc6a5011721ffe8eb1db36d348e0df866e428c31ae3e2c103f8c88.scope/kubepods/burstable/podaae2b796-aecd-4431-88ac-d979d4e06784", + "entity": null, + "mcp_url": "http://127.0.0.1:18600/mcp", + "reset": "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold/reset-mcp.sh", + "sampling": "agent_qwen38", + "cap_s": 240.0, + "guard_s": 99999.0, + "settle_s": 30.0, + "no_warmup": false, + "no_rapl": false, + "quiesce_s": 300.0, + "warmup_timeout_s": 600.0, + "baseline": "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs/baseline-hold1.jsonl", + "level": "typical", + "pre_s": 10.0 + }, + "host": { + "cpu": "Intel(R) Xeon(R) 6741P", + "kernel": "7.0.0-31-generic", + "loadavg": "9.29 3.97 2.99 22/5228 1564302", + "python": "3.14.4" + }, + "lock": { + "schema": "k3d-gpu.agent-bench-lock/v1", + "read": "2026-10-17", + "note": "Every external piece of the agent benchmark, pinned. Image digests: Docker Hub tag API (hub.docker.com/v2/repositories//tags/) or the repo's own lock files; Hugging Face revisions: huggingface.co/api/models/; Ollama: sha256 of the registry manifest body (registry.ollama.ai/v2/library/qwen3/manifests/, Accept application/vnd.docker.distribution.manifest.v2+json).", + "suites": { + "bfcl_eval": { + "version": "2025.12.17", + "wheel": "bfcl_eval-2025.12.17-py3-none-any.whl", + "wheel_sha256": "8555bc9407a56682ceb7d969e87eb724f6b679deb0ef05114d9c6e786406b103", + "pypi": "https://pypi.org/project/bfcl-eval/2025.12.17/", + "gorilla_commit": "f7cf7359b7ac615a0b294831c5ba2bc95ee4a000", + "gorilla_commit_url": "https://github.com/ShishirPatil/gorilla/commit/f7cf7359b7ac615a0b294831c5ba2bc95ee4a000", + "install": "uv pip install --python /bin/python --require-hashes -r bench/eatflow/bfcl-constraints.txt (every package pinned with its sha256, generated by uv pip compile --generate-hashes; the bfcl-eval wheel hash is the one above)", + "note": "PyPI also has 2025.12.17.1 and 2026.3.23; do not float." + }, + "bfcl_board": { + "url": "https://gorilla.cs.berkeley.edu/data_overall.csv", + "sha256_on_read": "ede5081a24bc5ac7f9b3b09a52af6bcfcd35aa0f1709239fed979db77a8159a6", + "row": "Qwen3-4B-Instruct-2507 (FC)", + "values": { + "overall": 35.68, + "non_live_simple": 75.5, + "multiple": 93.5, + "parallel": 92.5, + "parallel_multiple": 90.0, + "live_simple": 79.07, + "live_multiple": 76.16, + "irrelevance": 84.93 + }, + "handler": "bfcl_eval/constants/model_config.py: 'Qwen/Qwen3-4B-Instruct-2507-FC' -> QwenFCHandler (local_inference, /v1/completions, BFCL applies the chat template)" + }, + "blueprints": { + "repo": "https://github.com/intel/enterprise-ai-blueprints", + "tag": "v1.0.0", + "commit": "758d4845520ca5456bf732264ef02d92a26d25a6", + "banking_data_py_sha256": "37b7b9c84af4395c370c4ac2d3ca063b0b4f9d0d2054c082013f9ea6e3e13916", + "banking_server_py_sha256": "1edb561a599a7879330e4e83b5005825ba4090d928401bafb3c0c66e083b8e78", + "coding_agent_py_sha256": "c30abfe159aa35d70b37572acd4b55a0a89d31187480add3c73a1cacc2de5991", + "agent_framework": "agent-framework-core==1.19.0, agent-framework-openai==1.14.4, agent-framework-devui==1.0.0b260918, mcp==1.30.0 (usecases/banking-assistant/src/requirements.txt)" + }, + "eat": { + "repo": "https://github.com/intel/enterprise-agent-toolkit", + "ref": "release-1.0.3-hotfix", + "commit": "0d3438d7119de44cafc5903f66665c0b623e3333" + }, + "coding_agent": { + "k8s_agent_sandbox": "1.0.7", + "wheel": "k8s_agent_sandbox-1.0.7-py3-none-any.whl", + "wheel_sha256": "1c0b92044b1612c5fb65b95e85558fd6633b4c64b6e1cfb624ef8c4ca49f5c5c", + "agent_sandbox": "v1.0.7 (share/agent-sandbox.yml, k3d-gpu sandbox up)", + "constraints": "bench/eatflow/coding-constraints.txt", + "install": "uv pip install --python /bin/python --require-hashes -r bench/eatflow/coding-constraints.txt (Python 3.12; every package with its sha256)", + "note": "H43 amendment W1: the blueprint at 758d484 pins k8s-agent-sandbox 0.4.6 (v1alpha1 claims), which agent-sandbox v1.0.7 does not serve; 1.0.7 runs behind bench/eatflow/coding/sitecustomize.py. The router is bench/eatflow/sbrouter.py on the proxy_python image." + } + }, + "models": { + "target": { + "repo": "Qwen/Qwen3-4B-Instruct-2507", + "revision": "cdbee75f17c01a7cc42f958dc650907174af0554" + }, + "awq": { + "repo": "cyankiwi/Qwen3-4B-Instruct-2507-AWQ-4bit", + "revision": "c46e2277ae3dee7d030f45c58987b291edc5c1d6", + "note": "third-party quant: confound recorded" + }, + "gguf": { + "repo": "unsloth/Qwen3-4B-Instruct-2507-GGUF", + "revision": "a06e946bb6b655725eafa393f4a9745d460374c9", + "file": "Qwen3-4B-Instruct-2507-Q4_K_M.gguf", + "lfs_sha256": "3605803b982cb64aead44f6c1b2ae36e3acdb41d8e46c8a94c6533bc4c67e597", + "bytes": 2497281120 + }, + "ollama": { + "tag": "qwen3:4b-instruct-2507-q4_K_M", + "manifest_sha256": "0edcdef34593eac1aa2be9c7d06c432dcf81945adca5eca2f27662c18f168ba0", + "model_blob": "sha256:85e4a5b7b8ef0e48af0e8658f5aaab9c2324c76c1641493f4d1e25fce54b18b9" + }, + "draft": { + "repo": "OpenVINO/Qwen3-0.6B-int4-ov", + "revision": "f864c6106efb6c7f7b4ef274a78a98e37210dddd" + }, + "openvino_int4": { + "note": "No OpenVINO int4 export of Qwen3-4B-Instruct-2507 is published; A1/A2 use a self-export (optimum-intel, NNCF int4 sym group 128) whose xml/bin sha256 is recorded in the run's facts.json." + } + }, + "images": { + "ovms": { + "ref": "openvino/model_server:2026.4.1-gpu@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58", + "source": "images.lock / bench/competitors.lock" + }, + "vllm_cpu_eat": { + "ref": "vllm/vllm-openai-cpu:v0.24.0@sha256:b0e6f8604413a8bfcca8836bb4b57b47f1aa653957e07f8903ece531309981a1", + "source": "integrations/eat/vllm-values.yaml; Docker Hub tag v0.24.0 digest" + }, + "vllm_cuda": { + "ref": "vllm/vllm-openai@sha256:a4a4c0437bf7240089da5f08aa370c4aee17ae5290f7a3b468825ee26c4c3a6b", + "source": "bench/apb/arm.sh; Docker Hub tag v0.31.0, linux/amd64 manifest (the tag's index digest is sha256:c1c9f6fd5c109ba7f0546a59f5b2f15fb87f64c77782e90a27b648b42a8e67c3)" + }, + "ollama": { + "ref": "ollama/ollama:0.40.5@sha256:5d3a2f25f9825628daba94dc0b1d421c7d57d6c03ae5faa9ba050e058bab8241", + "source": "bench/competitors.lock; Docker Hub tag 0.40.5 digest" + }, + "llamacpp": { + "ref": "ghcr.io/ggml-org/llama.cpp:server-b11706@sha256:f41d47eee678c02656eefc9ed8cce76cab584aa7032805eeef0e6639fcfda485", + "source": "bench/competitors.lock" + }, + "stress_ng": { + "ref": "ghcr.io/colinianking/stress-ng:72a9d0cae0f420a887991d4024d3101c4af7a4fc@sha256:2ea9a7c3828eda1b1ca3d0b0c5a669ec252f9caf97a6a33aedcea6b2a66d1cbe", + "source": "bench/load/load.py" + }, + "proxy_python": { + "ref": "python:3.14.8-slim@sha256:fbecd461fc72d21b2143c6eaba25143738079154a6d0ea3da3aec724f50eed4a", + "source": "docs-build/Dockerfile" + }, + "redis": { + "ref": "redis:8.10.2-alpine@sha256:9f8d2d789b9a1d1e3f283f979aa9cbf393373806472e1ccd460e2f9c869b833b", + "source": "H43 (amendment W1): the coding agent's memory store; Docker Hub tag 8.10.2-alpine index digest, read 2026-10-28; bench/eatflow/live.sh REDIS_IMAGE" + } + }, + "sampling": { + "_note": "Pinned by the recording proxy into every agent LLM request (it replaces every sampling key the agent sent), so no engine default applies. agent_local: greedy, every local arm. agent_remote: arm R through harvest, which forbids temperature 0 and logprobs; top_k 1 is greedy there; GLM-5.3 always reasons, so reasoning_effort low (the operator's known-good harvest client setting) keeps the 256-token cap for the answer. BFCL keeps its own temperature 0.001 (the board's setting).", + "agent_local": { + "temperature": 0, + "top_p": 1, + "max_tokens": 1024 + }, + "agent_remote": { + "top_k": 1, + "max_tokens": 256, + "reasoning_effort": "low" + }, + "agent_gemma4": { + "_source": "crescent amendment M1: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized @ f1e06dc5, README 'Sampling Parameters' and generation_config.json; thinking off: the template default (enable_thinking false) sent explicitly, as REFERENCE.md 1.1 gives it per request", + "temperature": 1.0, + "top_p": 0.95, + "top_k": 64, + "max_tokens": 1024, + "chat_template_kwargs": { + "enable_thinking": false + } + }, + "agent_qwen38": { + "_source": "H53: Qwen/Qwen3.8-27B @ 1d4bf0f2 README 'Best Practices', instruct (non-thinking) mode; min_p 0.0 and repetition_penalty 1.0 are the engines' defaults and are not sent; thinking off by the card's chat_template_kwargs", + "temperature": 0.7, + "top_p": 0.8, + "top_k": 20, + "presence_penalty": 1.5, + "max_tokens": 1024, + "chat_template_kwargs": { + "enable_thinking": false + } + } + }, + "hypotheses": { + "H40": { + "workload": "BFCL-S100 board", + "arms": [ + "A3ref" + ], + "levels": [ + "none" + ], + "order": "single" + }, + "H41": { + "workload": "BFCL-S100 server-fc", + "arms": [ + "A3ref", + "A1", + "A3", + "B1", + "C1", + "C2" + ], + "levels": [ + "typical" + ], + "order": "ABBA" + }, + "H42": { + "workload": "B24 N=1", + "arms": [ + "A1", + "A2", + "A3", + "B1", + "C1", + "C2" + ], + "levels": [ + "typical" + ], + "order": "ABBA" + }, + "H43": { + "workload": "C12 N=1", + "arms": [ + "A1", + "A3", + "B1" + ], + "levels": [ + "typical", + "heavy" + ], + "order": "ABBA" + }, + "H44": { + "workload": "B24 first 12 tasks N=1", + "arms": [ + "A1", + "B1" + ], + "levels": [ + "none", + "heavy" + ], + "order": "alternate" + }, + "H45": { + "workload": "B15-RO ramp N=1,2,4,8", + "arms": [ + "A1", + "A3", + "C1", + "C2" + ], + "levels": [ + "typical" + ], + "order": "ramp" + }, + "H46": { + "workload": "H42 blocks (energy) + one 120 s load-only control window per arm", + "arms": [ + "A1", + "A2", + "A3", + "B1", + "C1", + "C2" + ], + "levels": [ + "typical" + ], + "order": "control" + }, + "H47": { + "workload": "B24 N=1", + "arms": [ + "A4", + "A1" + ], + "levels": [ + "typical" + ], + "order": "ABBA", + "precondition": "H34 SUPPORTED" + }, + "H48": { + "workload": "4 B24 tasks + 8 S100 ids, <= 20 LLM requests", + "arms": [ + "R" + ], + "levels": [ + "none" + ], + "order": "single", + "max_requests": 20 + } + } + }, + "ts": 1793319592.5549145, + "hold": 1, + "arm": "AG" +} \ No newline at end of file diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/proxy-AG-hold1.jsonl b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/proxy-AG-hold1.jsonl new file mode 100644 index 00000000..77c131b2 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/proxy-AG-hold1.jsonl @@ -0,0 +1,2 @@ +{"ts_send": 1793319592.8474762, "path": "/v1/chat/completions", "stream": true, "model": "Qwen/Qwen3.8-27B", "last_user": "My user id is U1001. Which account ids do I have? Answer with the ids only.", "first_user": "My user id is U1001. Which account ids do I have? Answer with the ids only.", "user_msgs": 1, "tool_results": [], "sampling": {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5, "max_tokens": 1024, "chat_template_kwargs": {"enable_thinking": false}}, "tools": ["get_user", "find_user", "get_accounts", "get_account_balance", "get_balances", "get_transactions", "get_loans", "get_loan_summary", "transfer_funds", "pay_loan"], "status": 200, "error": null, "ttft_s": 2.946187, "e2e_s": 3.673176, "content": "", "tool_calls": [{"name": "get_accounts", "arguments": "{\"user_id\":\"U1001\"}", "valid": true, "reason": "ok"}], "tool_source": "structured", "tool_valid": true, "finish_reason": "tool_calls", "error_body": null, "usage": {"completion_tokens": 30, "prompt_tokens": 1929, "total_tokens": 1959}, "prompt_tokens": 1929, "completion_tokens": 30, "cached_tokens": null, "tag": null} +{"ts_send": 1793319596.531258, "path": "/v1/chat/completions", "stream": true, "model": "Qwen/Qwen3.8-27B", "last_user": "My user id is U1001. Which account ids do I have? Answer with the ids only.", "first_user": "My user id is U1001. Which account ids do I have? Answer with the ids only.", "user_msgs": 1, "tool_results": ["{\n \"user_id\": \"U1001\",\n \"count\": 2,\n \"accounts\": [\n {\n \"account_id\": \"ACC-1001-CHK\",\n \"user_id\": \"U1001\",\n \"type\": \"checking\",\n \"nickname\": \"Everyday Checking\",\n \"currency\": \"USD\",\n \"opened_date\": \"2016-03-12\",\n \"status\": \"active\"\n },\n {\n \"account_id\": \"ACC-1001-SAV\",\n \"user_id\": \"U1001\",\n \"type\": \"savings\",\n \"nickname\": \"Rainy Day Savings\",\n \"currency\": \"USD\",\n \"opened_date\": \"2017-01-05\",\n \"status\": \"active\"\n }\n ]\n}"], "sampling": {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5, "max_tokens": 1024, "chat_template_kwargs": {"enable_thinking": false}}, "tools": ["get_user", "find_user", "get_accounts", "get_account_balance", "get_balances", "get_transactions", "get_loans", "get_loan_summary", "transfer_funds", "pay_loan"], "status": 200, "error": null, "ttft_s": 0.370188, "e2e_s": 1.521517, "content": "ACC-1001-CHK, ACC-1001-SAV\n\n\nACC-1001-CHK, ACC-1001-SAV", "tool_calls": [], "tool_source": "none", "tool_valid": null, "finish_reason": "stop", "error_body": null, "usage": {"completion_tokens": 38, "prompt_tokens": 2185, "total_tokens": 2223, "completion_tokens_details": {"reasoning_tokens": 0}}, "prompt_tokens": 2185, "completion_tokens": 38, "cached_tokens": null, "tag": null} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/registration.json b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/registration.json new file mode 100644 index 00000000..94baebcd --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/registration.json @@ -0,0 +1 @@ +{"hyp": "H53", "rule": "L3", "seed": "H53-crescent-2026-10-20"} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/warmups.jsonl b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/warmups.jsonl new file mode 100644 index 00000000..a02039a4 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-AG/pairs/warmups.jsonl @@ -0,0 +1 @@ +{"hold": 1, "arm": "AG", "seconds": 5.476, "error": null, "answer": "ACC-1001-CHK, ACC-1001-SAV\n\n\nACC-1001-CHK, ACC-1001-SAV"} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/b-listing.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/b-listing.txt new file mode 100644 index 00000000..de902c4d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/b-listing.txt @@ -0,0 +1,34 @@ +.gitattributes 1570 1793310278.7959648510 +LICENSE 11544 1793310278.8089651410 +README.md 65012 1793310278.8199653860 +chat_template.jinja 8952 1793310278.8139652520 +config.json 4312 1793310278.8129652300 +crc32.txt 238 1793310278.8299656090 +generation_config.json 202 1793310278.8259655200 +merges.txt 3353259 1793310278.8589662550 +model-00001-of-00018.safetensors 3966730552 1793310507.8016561670 +model-00002-of-00018.safetensors 3043080328 1793310511.8507358520 +model-00003-of-00018.safetensors 2542796952 1793310459.7537105830 +model-00004-of-00018.safetensors 3988973152 1793310571.2509048500 +model-00005-of-00018.safetensors 2099339864 1793310430.8621419970 +model-00006-of-00018.safetensors 3979553696 1793310509.6426923980 +model-00007-of-00018.safetensors 2108759344 1793310436.7122571270 +model-00008-of-00018.safetensors 3979553696 1793310573.9919587940 +model-00009-of-00018.safetensors 2108759344 1793310591.7193076680 +model-00010-of-00018.safetensors 3979553696 1793310721.1158541950 +model-00011-of-00018.safetensors 2108759344 1793310632.3041063770 +model-00012-of-00018.safetensors 3979553696 1793310730.4580380490 +model-00013-of-00018.safetensors 2108759344 1793310660.3256578430 +model-00014-of-00018.safetensors 3979553696 1793310752.6844754650 +model-00015-of-00018.safetensors 2108759344 1793310664.6107421730 +model-00016-of-00018.safetensors 3979564040 1793310758.3635872300 +model-00017-of-00018.safetensors 2108759344 1793310689.8802394770 +model-00018-of-00018.safetensors 3392197344 1793310751.2064463780 +model.safetensors.index.json 112216 1793310660.4686606570 +preprocessor_config.json 390 1793310660.5806628610 +tokenizer.json 12809320 1793310661.7056850020 +tokenizer_config.json 17928 1793310661.8266873830 +video_preprocessor_config.json 385 1793310661.9656901190 +vocab.json 6722759 1793310662.3626979320 +77042094076611b69791a610065f28b7013b8c621795fa86ddccc8bac7d1b9df model.safetensors.index.json +191e0af232104ed8b65258cf3fb2b842e288008baca7633c11b82a1ac7203aab config.json diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/blueprint-commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/blueprint-commit.txt new file mode 100644 index 00000000..2cf3b989 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/blueprint-commit.txt @@ -0,0 +1 @@ +758d4845520ca5456bf732264ef02d92a26d25a6 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/commit.txt new file mode 100644 index 00000000..4af4a96a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/commit.txt @@ -0,0 +1 @@ +1f39488be7695dc0d1848347bf1c91f04025bc55 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/deploy.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/deploy.yaml new file mode 100644 index 00000000..664aad4b --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/deploy.yaml @@ -0,0 +1,189 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: vllm-qwen38-27b-xpu-config + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +data: + http_proxy: '' + https_proxy: '' + no_proxy: '' + TORCHINDUCTOR_CACHE_DIR: /tmp + NUMBA_CACHE_DIR: /tmp + HF_HOME: /data + OUTLINES_CACHE_DIR: /tmp/.cache/outlines + VLLM_NO_USAGE_STATS: '1' + DO_NOT_TRACK: '1' + CCL_ALLREDUCE: ring + CCL_PROCESS_LAUNCHER: none + CCL_WORKER_AFFINITY: '' + CCL_WORKER_OFFLOAD: '0' + CCL_ZE_IPC_EXCHANGE: pidfd + HF_ENDPOINT: https://huggingface.co + HF_HUB_DISABLE_XET: '1' + HF_HUB_ENABLE_HF_TRANSFER: '0' + HOME: /tmp + LOGNAME: vllm + ONEAPI_DEVICE_SELECTOR: level_zero:* + SYCL_CACHE_PERSISTENT: '1' + TORCH_LLM_ALLREDUCE: '1' + TRITON_CACHE_DIR: /tmp/.triton + TRITON_HOME: /tmp/.triton + USER: vllm + VLLM_ALLOW_LONG_MAX_MODEL_LEN: '1' + VLLM_ATTENTION_BACKEND: TORCH_SDPA + VLLM_TARGET_DEVICE: xpu + VLLM_USE_V1: '1' + VLLM_WORKER_MULTIPROC_METHOD: spawn + XDG_CACHE_HOME: /tmp/.cache + ZES_ENABLE_SYSMAN: '1' + ZE_ENABLE_PCI_ID_DEVICE_ORDER: '1' + ZE_FLAT_DEVICE_HIERARCHY: FLAT + HF_HUB_OFFLINE: '1' +--- +apiVersion: v1 +kind: Service +metadata: + name: vllm-qwen38-27b-xpu-service + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 8000 + protocol: TCP + name: vllm + selector: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-qwen38-27b-xpu + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +spec: + selector: + matchLabels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + template: + metadata: + annotations: null + labels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + spec: + securityContext: + fsGroup: 1001 + supplementalGroups: + - 990 + - 44 + containers: + - name: vllm + envFrom: + - configMapRef: + name: vllm-qwen38-27b-xpu-config + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - SYS_NICE + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: false + runAsUser: 0 + seccompProfile: + type: RuntimeDefault + image: vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + imagePullPolicy: IfNotPresent + command: + - bash + - -c + - source /opt/intel/oneapi/setvars.sh --force && python3 -m vllm.entrypoints.openai.api_server + "$@" + - -- + args: + - --host + - 0.0.0.0 + - --dtype + - float16 + - --block-size + - '64' + - --gpu-memory-utilization + - '0.9' + - --enforce-eager + - --no-enable-prefix-caching + - --trust-remote-code + - --disable-sliding-window + - --enable-auto-tool-choice + - --tool-call-parser + - qwen3_xml + - --tensor-parallel-size + - '4' + - --model + - /data/Qwen3.8-27B + - --served-model-name + - Qwen/Qwen3.6-27B + - --port + - '8000' + volumeMounts: + - mountPath: /data + name: model-volume + - mountPath: /dev/shm + name: shm + - mountPath: /tmp + name: tmp + - mountPath: /root/.cache/vllm + name: vllm-cache + ports: + - name: http + containerPort: 8000 + protocol: TCP + readinessProbe: + failureThreshold: 200 + httpGet: + path: /health + port: http + initialDelaySeconds: 10 + periodSeconds: 5 + resources: + limits: + gpu.intel.com/xe: '4' + volumes: + - name: model-volume + hostPath: + path: /var/apb-ro + type: Directory + - name: shm + emptyDir: + medium: Memory + sizeLimit: 1Gi + - name: tmp + emptyDir: {} + - name: vllm-cache + emptyDir: {} + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: ei-inference-eligible + operator: In + values: + - 'true' diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/describe-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/describe-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt new file mode 100644 index 00000000..d223e436 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/describe-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt @@ -0,0 +1,119 @@ +Name: vllm-qwen38-27b-xpu-799dc85d7c-ng77g +Namespace: default +Priority: 0 +Service Account: default +Node: k3d-k3dgpu-e-h53-server-0/172.19.0.2 +Start Time: Fri, 30 Oct 2026 00:23:06 +0000 +Labels: app.kubernetes.io/instance=vllm-qwen38-27b-xpu + app.kubernetes.io/name=vllm + pod-template-hash=799dc85d7c +Annotations: +Status: Running +IP: 10.42.0.14 +IPs: + IP: 10.42.0.14 +Controlled By: ReplicaSet/vllm-qwen38-27b-xpu-799dc85d7c +Containers: + vllm: + Container ID: containerd://bf0eaf47310725fbdc8ba34fe3d4ed8108c6ae783b775c4d753d6fc589705d8f + Image: vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + Image ID: docker.io/vllm/vllm-openai-xpu@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + Port: 8000/TCP (http) + Host Port: 0/TCP (http) + SeccompProfile: RuntimeDefault + Command: + bash + -c + source /opt/intel/oneapi/setvars.sh --force && python3 -m vllm.entrypoints.openai.api_server "$@" + -- + Args: + --host + 0.0.0.0 + --dtype + float16 + --block-size + 64 + --gpu-memory-utilization + 0.9 + --enforce-eager + --no-enable-prefix-caching + --trust-remote-code + --disable-sliding-window + --enable-auto-tool-choice + --tool-call-parser + qwen3_xml + --tensor-parallel-size + 4 + --model + /data/Qwen3.8-27B + --served-model-name + Qwen/Qwen3.6-27B + --port + 8000 + State: Waiting + Reason: CrashLoopBackOff + Last State: Terminated + Reason: Error + Exit Code: 1 + Started: Fri, 30 Oct 2026 00:25:08 +0000 + Finished: Fri, 30 Oct 2026 00:25:08 +0000 + Ready: False + Restart Count: 2 + Limits: + gpu.intel.com/xe: 4 + Requests: + gpu.intel.com/xe: 4 + Readiness: http-get http://:http/health delay=10s timeout=1s period=5s successThreshold=1 failureThreshold=200 + Environment Variables from: + vllm-qwen38-27b-xpu-config ConfigMap Optional: false + Environment: + Mounts: + /data from model-volume (rw) + /dev/shm from shm (rw) + /root/.cache/vllm from vllm-cache (rw) + /tmp from tmp (rw) + /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jxpk9 (ro) +Conditions: + Type Status + PodReadyToStartContainers True + Initialized True + Ready False + ContainersReady False + PodScheduled True +Volumes: + model-volume: + Type: HostPath (bare host directory volume) + Path: /var/apb-ro + HostPathType: Directory + shm: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: Memory + SizeLimit: 1Gi + tmp: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: + vllm-cache: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: + kube-api-access-jxpk9: + Type: Projected (a volume that contains injected data from multiple sources) + TokenExpirationSeconds: 3607 + ConfigMapName: kube-root-ca.crt + Optional: false + DownwardAPI: true +QoS Class: BestEffort +Node-Selectors: +Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s + node.kubernetes.io/unreachable:NoExecute op=Exists for 300s +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 2m12s default-scheduler Successfully assigned default/vllm-qwen38-27b-xpu-799dc85d7c-ng77g to k3d-k3dgpu-e-h53-server-0 + Normal Pulling 2m12s kubelet spec.containers{vllm}: Pulling image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" + Normal Pulled 28s kubelet spec.containers{vllm}: Successfully pulled image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" in 1m43.931s (1m43.931s including waiting). Image size: 4143776505 bytes. + Normal Created 11s (x3 over 28s) kubelet spec.containers{vllm}: Container created + Normal Started 11s (x3 over 28s) kubelet spec.containers{vllm}: Container started + Normal Pulled 11s (x2 over 26s) kubelet spec.containers{vllm}: Container image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" already present on machine and can be accessed by the pod + Warning BackOff 10s (x4 over 25s) kubelet spec.containers{vllm}: Back-off restarting failed container vllm in pod vllm-qwen38-27b-xpu-799dc85d7c-ng77g_default(6077fde9-5952-450e-b869-3aa47c254b6a) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/eat.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/eat.yaml new file mode 100644 index 00000000..7c2d0407 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/eat.yaml @@ -0,0 +1,301 @@ +--- +# Source: vllm/templates/secret.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: Secret +metadata: + name: vllm-qwen38-27b-xpu-secret + labels: + app: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +type: Opaque +data: +--- +# Source: vllm/templates/configmap.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: ConfigMap +metadata: + name: vllm-qwen38-27b-xpu-config + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +data: + http_proxy: "" + https_proxy: "" + no_proxy: "" + TORCHINDUCTOR_CACHE_DIR: "/tmp" + NUMBA_CACHE_DIR: "/tmp" + HF_HOME: "/data" + OUTLINES_CACHE_DIR: "/tmp/.cache/outlines" + VLLM_NO_USAGE_STATS: "1" + DO_NOT_TRACK: "1" + CCL_ALLREDUCE: "ring" + CCL_PROCESS_LAUNCHER: "none" + CCL_WORKER_AFFINITY: "" + CCL_WORKER_OFFLOAD: "0" + CCL_ZE_IPC_EXCHANGE: "pidfd" + DO_NOT_TRACK: "1" + HF_ENDPOINT: "https://huggingface.co" + HF_HUB_DISABLE_XET: "1" + HF_HUB_ENABLE_HF_TRANSFER: "0" + HOME: "/tmp" + LOGNAME: "vllm" + ONEAPI_DEVICE_SELECTOR: "level_zero:*" + SYCL_CACHE_PERSISTENT: "1" + TORCH_LLM_ALLREDUCE: "1" + TRITON_CACHE_DIR: "/tmp/.triton" + TRITON_HOME: "/tmp/.triton" + USER: "vllm" + VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1" + VLLM_ATTENTION_BACKEND: "TORCH_SDPA" + VLLM_NO_USAGE_STATS: "1" + VLLM_TARGET_DEVICE: "xpu" + VLLM_USE_V1: "1" + VLLM_WORKER_MULTIPROC_METHOD: "spawn" + XDG_CACHE_HOME: "/tmp/.cache" + ZES_ENABLE_SYSMAN: "1" + ZE_ENABLE_PCI_ID_DEVICE_ORDER: "1" + ZE_FLAT_DEVICE_HIERARCHY: "FLAT" +--- +# Source: vllm/templates/pvc.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: vllm-qwen38-27b-xpu-pvc + namespace: default +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 250Gi # Adjust this according to model storage size +--- +# Source: vllm/templates/service.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: v1 +kind: Service +metadata: + name: vllm-qwen38-27b-xpu-service + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 8000 + protocol: TCP + name: vllm + selector: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu +--- +# Source: vllm/templates/deployment.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-qwen38-27b-xpu + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +spec: + selector: + matchLabels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + template: + metadata: + annotations: + labels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + spec: + # + # serviceAccountName: default + securityContext: + fsGroup: 1001 + supplementalGroups: + - 990 + - 44 + containers: + - name: vllm + envFrom: + - configMapRef: + name: vllm-qwen38-27b-xpu-config + - secretRef: + name: vllm-qwen38-27b-xpu-secret + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - SYS_NICE + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: false + runAsUser: 0 + seccompProfile: + type: RuntimeDefault + image: "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" + imagePullPolicy: IfNotPresent + command: + - bash + - -c + - source /opt/intel/oneapi/setvars.sh --force && export LD_LIBRARY_PATH=/host-gpu-libs:${LD_LIBRARY_PATH} + && python3 -m vllm.entrypoints.openai.api_server "$@" + - -- + # command: + # - /bin/bash + # - -c + # - | + # python3 -m vllm.entrypoints.openai.api_server --dtype float16 --model Qwen/Qwen3.6-27B --port 8000 --tensor-parallel-size 4 --block-size 64 --max-model-len 32768 --disable-log-requests + args: + - "--host" + - "0.0.0.0" + - "--dtype" + - "float16" + - "--block-size" + - "64" + - "--gpu-memory-utilization" + - "0.9" + - "--enforce-eager" + - "--no-enable-prefix-caching" + - "--trust-remote-code" + - "--disable-sliding-window" + - "--enable-auto-tool-choice" + - "--tool-call-parser" + - "qwen3_xml" + - "--tensor-parallel-size" + - "4" + - "--model" + - "Qwen/Qwen3.6-27B" + - --served-model-name + - "Qwen/Qwen3.6-27B" + - "--port" + - "8000" + volumeMounts: + - mountPath: /data + name: model-volume + - mountPath: /dev/shm + name: shm + - mountPath: /tmp + name: tmp + - mountPath: /root/.cache/vllm + name: vllm-cache + - mountPath: /host-gpu-libs + name: host-gpu-libs + readOnly: true + - mountPath: /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1 + name: host-gpu-libs + readOnly: true + subPath: libze_intel_gpu.so.1 + - mountPath: /usr/lib/x86_64-linux-gnu/libze_loader.so.1 + name: host-gpu-libs + readOnly: true + subPath: libze_loader.so.1 + - mountPath: /etc/OpenCL/vendors + name: host-opencl-vendors + readOnly: true + - mountPath: /usr/lib/x86_64-linux-gnu/intel-opencl + name: host-intel-opencl + readOnly: true + ports: + - name: http + containerPort: 8000 + protocol: TCP + readinessProbe: + failureThreshold: 98 + httpGet: + path: /health + port: http + initialDelaySeconds: 120 + periodSeconds: 30 + resources: + limits: + "gpu.intel.com/xe": "4" + + volumes: + - name: model-volume + persistentVolumeClaim: + claimName: vllm-qwen38-27b-xpu-pvc + - name: shm + emptyDir: + medium: Memory + sizeLimit: 1Gi + - name: tmp + emptyDir: {} + - name: vllm-cache + emptyDir: {} + - hostPath: + path: /usr/lib/x86_64-linux-gnu + type: Directory + name: host-gpu-libs + - hostPath: + path: /etc/OpenCL/vendors + type: Directory + name: host-opencl-vendors + - hostPath: + path: /usr/lib/x86_64-linux-gnu/intel-opencl + type: Directory + name: host-intel-opencl + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: ei-inference-eligible + operator: In + values: + - "true" + +--- +# Source: vllm/templates/apisixroutes.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +--- +# Source: vllm/templates/chat-template-configmap.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +--- +# Source: vllm/templates/ingress.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# Please edit the object below. Lines beginning with a '#' will be ignored, +# and an empty file will abort the edit. If an error occurs while saving this file will be +# reopened with the relevant failures. +--- +# Source: vllm/templates/ingress_eks.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +--- +# Source: vllm/templates/route.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +--- +# Source: vllm/templates/servicemonitor.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/events.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/events.txt new file mode 100644 index 00000000..527d14ff --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/events.txt @@ -0,0 +1,163 @@ +NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE +default 4m21s Normal Scheduled pod/gpu-hold-2 Successfully assigned default/gpu-hold-2 to k3d-k3dgpu-e-h53-server-0 +default 4m21s Normal Scheduled pod/gpu-hold-3 Successfully assigned default/gpu-hold-3 to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 4m47s Normal Scheduled pod/nfd-master-74b6cfb7df-m4whr Successfully assigned node-feature-discovery/nfd-master-74b6cfb7df-m4whr to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 4m47s Normal Scheduled pod/nfd-gc-7d6955597d-v5ns2 Successfully assigned node-feature-discovery/nfd-gc-7d6955597d-v5ns2 to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/local-path-provisioner-6858d854cf-8t662 Successfully assigned kube-system/local-path-provisioner-6858d854cf-8t662 to k3d-k3dgpu-e-h53-server-0 +default 2m12s Normal Scheduled pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Successfully assigned default/vllm-qwen38-27b-xpu-799dc85d7c-ng77g to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/intel-gpu-plugin-hxvpb Successfully assigned kube-system/intel-gpu-plugin-hxvpb to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/metrics-server-778c4649b4-l4d4j Successfully assigned kube-system/metrics-server-778c4649b4-l4d4j to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 4m47s Normal Scheduled pod/nfd-worker-87d64 Successfully assigned node-feature-discovery/nfd-worker-87d64 to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/helm-install-gateway-api-crd-gfhvl Successfully assigned kube-system/helm-install-gateway-api-crd-gfhvl to k3d-k3dgpu-e-h53-server-0 +kube-system 4m47s Normal Scheduled pod/coredns-577d995dff-c89zz Successfully assigned kube-system/coredns-577d995dff-c89zz to k3d-k3dgpu-e-h53-server-0 +default 4m54s Normal Starting node/k3d-k3dgpu-e-h53-server-0 +default 4m21s Normal Scheduled pod/gpu-hold-4 Successfully assigned default/gpu-hold-4 to k3d-k3dgpu-e-h53-server-0 +default 4m21s Normal Scheduled pod/gpu-hold-1 Successfully assigned default/gpu-hold-1 to k3d-k3dgpu-e-h53-server-0 +default 4m54s Normal NodeHasNoDiskPressure node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasNoDiskPressure +default 4m54s Normal CertificateExpirationOK node/k3d-k3dgpu-e-h53-server-0 Node and Certificate Authority certificates managed by k3s are OK +default 4m54s Normal Starting node/k3d-k3dgpu-e-h53-server-0 Starting kubelet. +default 4m54s Warning InvalidDiskCapacity node/k3d-k3dgpu-e-h53-server-0 invalid capacity 0 on image filesystem +default 4m54s Normal NodeHasSufficientMemory node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasSufficientMemory +default 4m54s Normal NodeReady node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeReady +default 4m54s Normal NodeHasSufficientPID node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasSufficientPID +default 4m54s Normal NodeAllocatableEnforced node/k3d-k3dgpu-e-h53-server-0 Updated Node Allocatable limit across pods +default 4m52s Normal NodePasswordValidationComplete node/k3d-k3dgpu-e-h53-server-0 Deferred node password secret validation complete +kube-system 4m51s Normal AppliedManifest addon/gateway-api-crd Applied manifest at "/var/lib/rancher/k3s/server/manifests/gateway-api-crd.yaml" +kube-system 4m51s Normal ApplyingManifest addon/ccm Applying manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml" +kube-system 4m51s Normal AppliedManifest addon/ccm Applied manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml" +kube-system 4m51s Normal ApplyingManifest addon/gaudi-device-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/gaudi-device-plugin.yaml" +kube-system 4m51s Normal ApplyingManifest addon/coredns Applying manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml" +kube-system 4m51s Normal AppliedManifest addon/coredns Applied manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml" +kube-system 4m51s Normal ApplyingManifest addon/gateway-api-crd Applying manifest at "/var/lib/rancher/k3s/server/manifests/gateway-api-crd.yaml" +kube-system 4m50s Normal ApplyingManifest addon/metrics-apiservice Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" +kube-system 4m50s Normal ApplyingManifest addon/auth-delegator Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-qat-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-qat-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-qat-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-qat-plugin.yaml" +default 4m50s Normal Synced node/k3d-k3dgpu-e-h53-server-0 Node synced successfully +kube-system 4m50s Normal AppliedManifest addon/resource-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" +kube-system 4m50s Normal ApplyingManifest addon/local-storage Applying manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-npu-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-npu-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-iaa-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-iaa-plugin.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-iaa-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-iaa-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-gpu-wsl-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-wsl-plugin.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-gpu-wsl-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-wsl-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-gpu-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-plugin.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-gpu-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/local-storage Applied manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml" +kube-system 4m50s Normal ApplyingManifest addon/aggregated-metrics-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" +kube-system 4m50s Normal AppliedManifest addon/aggregated-metrics-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-npu-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-npu-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/auth-delegator Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" +kube-system 4m50s Normal ApplyingManifest addon/auth-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" +kube-system 4m50s Normal AppliedManifest addon/auth-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" +kube-system 4m50s Normal AppliedManifest addon/metrics-apiservice Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" +kube-system 4m50s Normal ApplyingManifest addon/metrics-server-deployment Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" +kube-system 4m50s Normal AppliedManifest addon/metrics-server-deployment Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" +kube-system 4m50s Normal DeletingManifest addon/intel-gpu-dra Deleting manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-dra.yaml" +kube-system 4m50s Normal AppliedManifest addon/intel-dsa-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-dsa-plugin.yaml" +kube-system 4m50s Normal ApplyingManifest addon/intel-dsa-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-dsa-plugin.yaml" +kube-system 4m50s Normal ApplyJob helmchart/gateway-api-crd Applying HelmChart from https://%{KUBERNETES_API}%/static/charts/gateway-api-crd-1.6.103.tgz using Job kube-system/helm-install-gateway-api-crd +kube-system 4m50s Normal ApplyingManifest addon/metrics-server-service Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" +kube-system 4m50s Normal AppliedManifest addon/gaudi-device-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/gaudi-device-plugin.yaml" +kube-system 4m50s Normal AppliedManifest addon/metrics-server-service Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" +kube-system 4m50s Normal ApplyingManifest addon/node-feature-discovery Applying manifest at "/var/lib/rancher/k3s/server/manifests/node-feature-discovery.yaml" +kube-system 4m50s Normal ApplyingManifest addon/resource-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" +kube-system 4m49s Normal Suspended job/helm-install-gateway-api-crd Job suspended +default 4m49s Normal RegisteredNode node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 event: Registered Node k3d-k3dgpu-e-h53-server-0 in Controller +kube-system 4m48s Normal SuccessfulCreate replicaset/metrics-server-778c4649b4 Created pod: metrics-server-778c4649b4-l4d4j +kube-system 4m48s Normal DeletingManifest addon/traefik Deleting manifest at "/var/lib/rancher/k3s/server/manifests/traefik.yaml" +kube-system 4m48s Normal ResumeJob helmchart/gateway-api-crd Resumed synced Job kube-system/helm-install-gateway-api-crd +kube-system 4m48s Normal SuccessfulCreate replicaset/coredns-577d995dff Created pod: coredns-577d995dff-c89zz +node-feature-discovery 4m48s Normal SuccessfulCreate daemonset/nfd-worker Created pod: nfd-worker-87d64 +node-feature-discovery 4m48s Warning FailedCreatePodSandBox pod/nfd-worker-87d64 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "665049125198f0eec35b75b738ea9f7e00faed2fa597064534ec212c60c89b91": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m48s Warning FailedCreatePodSandBox pod/helm-install-gateway-api-crd-gfhvl Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "37b3a49edc472b8ffa47c8e9cd3de8fbe51ecba1b2a89d20fe8e165ef54a6cdb": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 4m48s Normal ScalingReplicaSet deployment/nfd-master Scaled up replica set nfd-master-74b6cfb7df from 0 to 1 +node-feature-discovery 4m48s Normal SuccessfulCreate replicaset/nfd-master-74b6cfb7df Created pod: nfd-master-74b6cfb7df-m4whr +kube-system 4m48s Normal SuccessfulCreate replicaset/local-path-provisioner-6858d854cf Created pod: local-path-provisioner-6858d854cf-8t662 +node-feature-discovery 4m48s Normal ScalingReplicaSet deployment/nfd-gc Scaled up replica set nfd-gc-7d6955597d from 0 to 1 +node-feature-discovery 4m48s Normal SuccessfulCreate replicaset/nfd-gc-7d6955597d Created pod: nfd-gc-7d6955597d-v5ns2 +kube-system 4m48s Normal SuccessfulCreate job/helm-install-gateway-api-crd Created pod: helm-install-gateway-api-crd-gfhvl +kube-system 4m48s Normal Resumed job/helm-install-gateway-api-crd Job resumed +kube-system 4m48s Normal ScalingReplicaSet deployment/local-path-provisioner Scaled up replica set local-path-provisioner-6858d854cf from 0 to 1 +kube-system 4m48s Normal ScalingReplicaSet deployment/coredns Scaled up replica set coredns-577d995dff from 0 to 1 +kube-system 4m48s Normal AppliedManifest addon/runtimes Applied manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml" +kube-system 4m48s Normal ScalingReplicaSet deployment/metrics-server Scaled up replica set metrics-server-778c4649b4 from 0 to 1 +kube-system 4m48s Normal ApplyingManifest addon/runtimes Applying manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml" +kube-system 4m48s Warning FailedCreatePodSandBox pod/intel-gpu-plugin-hxvpb Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0b8e1ef825f5cf964109182847df4ec64752f3a83a9d675d98dd47e1e78e567b": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m48s Normal AppliedManifest addon/rolebindings Applied manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml" +kube-system 4m48s Normal ApplyingManifest addon/rolebindings Applying manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml" +kube-system 4m48s Normal AppliedManifest addon/nvidia-device-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml" +kube-system 4m48s Normal ApplyingManifest addon/nvidia-device-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml" +kube-system 4m48s Normal DeletingManifest addon/nri-balloons Deleting manifest at "/var/lib/rancher/k3s/server/manifests/nri-balloons.yaml" +kube-system 4m48s Normal AppliedManifest addon/node-feature-discovery Applied manifest at "/var/lib/rancher/k3s/server/manifests/node-feature-discovery.yaml" +kube-system 4m48s Normal SuccessfulCreate daemonset/intel-gpu-plugin Created pod: intel-gpu-plugin-hxvpb +kube-system 4m47s Warning FailedCreatePodSandBox pod/coredns-577d995dff-c89zz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ee35c4604fe63c0786db2ce1fa271557f8e4094f1d63a6b3b18a7ee8591d15a5": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m47s Warning FailedCreatePodSandBox pod/local-path-provisioner-6858d854cf-8t662 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b404afdd943e2e02af64bff258bd72ff4900fdfe152117f2e4d2d8fb3c9a4fad": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m47s Warning FailedCreatePodSandBox pod/metrics-server-778c4649b4-l4d4j Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e8d628ccf8e639859e8df7149b7be5f14fc617e75945c4d6406e2ab6c659505b": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 4m47s Warning FailedCreatePodSandBox pod/nfd-gc-7d6955597d-v5ns2 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8d497575b988adc57da6778427415eae95251b88ef75ff3d315f14ad3faf4ddc": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 4m47s Warning FailedCreatePodSandBox pod/nfd-master-74b6cfb7df-m4whr Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1ffbc73bc8e37dd138d5b283910fd2d5b7a176dd1dba68a00c2d024e22974cb2": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 4m34s Normal Pulling pod/metrics-server-778c4649b4-l4d4j Pulling image "rancher/mirrored-metrics-server:v0.9.0" +kube-system 4m34s Normal Pulling pod/intel-gpu-plugin-hxvpb Pulling image "intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33" +kube-system 4m34s Normal Pulling pod/local-path-provisioner-6858d854cf-8t662 Pulling image "rancher/local-path-provisioner:v0.0.37" +kube-system 4m34s Normal Pulling pod/coredns-577d995dff-c89zz Pulling image "rancher/mirrored-coredns-coredns:1.14.7" +node-feature-discovery 4m34s Normal Pulling pod/nfd-gc-7d6955597d-v5ns2 Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +node-feature-discovery 4m34s Normal Pulling pod/nfd-worker-87d64 Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +kube-system 4m34s Normal Pulling pod/helm-install-gateway-api-crd-gfhvl Pulling image "rancher/klipper-helm:v0.13.3-build20260727" +node-feature-discovery 4m32s Normal Pulling pod/nfd-master-74b6cfb7df-m4whr Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +kube-system 4m31s Normal Started pod/intel-gpu-plugin-hxvpb Container started +kube-system 4m31s Normal Created pod/intel-gpu-plugin-hxvpb Container created +kube-system 4m31s Normal Pulled pod/intel-gpu-plugin-hxvpb Successfully pulled image "intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33" in 2.652s (2.652s including waiting). Image size: 5970136 bytes. +node-feature-discovery 4m30s Normal Started pod/nfd-master-74b6cfb7df-m4whr Container started +node-feature-discovery 4m30s Normal Created pod/nfd-master-74b6cfb7df-m4whr Container created +node-feature-discovery 4m30s Normal Pulled pod/nfd-master-74b6cfb7df-m4whr Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 1.71s (1.71s including waiting). Image size: 85412382 bytes. +node-feature-discovery 4m30s Warning Unhealthy pod/nfd-master-74b6cfb7df-m4whr Readiness probe failed: Get "http://10.42.0.9:8080/healthz": dial tcp 10.42.0.9:8080: connect: connection refused +node-feature-discovery 4m30s Normal Started pod/nfd-gc-7d6955597d-v5ns2 Container started +node-feature-discovery 4m30s Normal Created pod/nfd-gc-7d6955597d-v5ns2 Container created +node-feature-discovery 4m30s Normal Pulled pod/nfd-gc-7d6955597d-v5ns2 Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 3.227s (3.227s including waiting). Image size: 85412382 bytes. +kube-system 4m30s Normal Created pod/local-path-provisioner-6858d854cf-8t662 Container created +kube-system 4m30s Normal Pulled pod/local-path-provisioner-6858d854cf-8t662 Successfully pulled image "rancher/local-path-provisioner:v0.0.37" in 4.181s (4.181s including waiting). Image size: 15066012 bytes. +node-feature-discovery 4m30s Normal Pulled pod/nfd-worker-87d64 Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 3.112s (3.112s including waiting). Image size: 85412382 bytes. +node-feature-discovery 4m30s Normal Created pod/nfd-worker-87d64 Container created +node-feature-discovery 4m30s Normal Started pod/nfd-worker-87d64 Container started +kube-system 4m29s Normal Created pod/helm-install-gateway-api-crd-gfhvl Container created +kube-system 4m29s Warning Unhealthy pod/metrics-server-778c4649b4-l4d4j Readiness probe failed: Get "https://10.42.0.3:10250/readyz": dial tcp 10.42.0.3:10250: connect: connection refused +kube-system 4m29s Normal Started pod/helm-install-gateway-api-crd-gfhvl Container started +kube-system 4m29s Normal Created pod/coredns-577d995dff-c89zz Container created +kube-system 4m29s Normal Pulled pod/helm-install-gateway-api-crd-gfhvl Successfully pulled image "rancher/klipper-helm:v0.13.3-build20260727" in 4.831s (4.831s including waiting). Image size: 74172006 bytes. +kube-system 4m29s Warning Unhealthy pod/coredns-577d995dff-c89zz Readiness probe failed: Get "http://10.42.0.2:8181/ready": dial tcp 10.42.0.2:8181: connect: connection refused +kube-system 4m29s Normal Pulled pod/metrics-server-778c4649b4-l4d4j Successfully pulled image "rancher/mirrored-metrics-server:v0.9.0" in 4.275s (4.275s including waiting). Image size: 23258428 bytes. +kube-system 4m29s Normal Pulled pod/coredns-577d995dff-c89zz Successfully pulled image "rancher/mirrored-coredns-coredns:1.14.7" in 4.936s (4.936s including waiting). Image size: 24261571 bytes. +kube-system 4m29s Normal Created pod/metrics-server-778c4649b4-l4d4j Container created +kube-system 4m29s Normal Started pod/local-path-provisioner-6858d854cf-8t662 Container started +kube-system 4m29s Normal Started pod/coredns-577d995dff-c89zz Container started +kube-system 4m29s Normal Started pod/metrics-server-778c4649b4-l4d4j Container started +kube-system 4m26s Normal Completed job/helm-install-gateway-api-crd Job completed +default 4m21s Normal Pulling pod/gpu-hold-4 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 4m21s Normal Pulling pod/gpu-hold-2 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 4m21s Normal Pulling pod/gpu-hold-1 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 4m21s Normal Pulling pod/gpu-hold-3 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 4m19s Normal Pulled pod/gpu-hold-4 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.802s (1.802s including waiting). Image size: 2222118 bytes. +default 4m19s Normal Created pod/gpu-hold-1 Container created +default 4m19s Normal Created pod/gpu-hold-4 Container created +default 4m19s Normal Started pod/gpu-hold-1 Container started +default 4m19s Normal Pulled pod/gpu-hold-2 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.941s (1.941s including waiting). Image size: 2222118 bytes. +default 4m19s Normal Started pod/gpu-hold-3 Container started +default 4m19s Normal Pulled pod/gpu-hold-3 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.88s (1.88s including waiting). Image size: 2222118 bytes. +default 4m19s Normal Created pod/gpu-hold-2 Container created +default 4m19s Normal Started pod/gpu-hold-2 Container started +default 4m19s Normal Pulled pod/gpu-hold-1 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 2.004s (2.004s including waiting). Image size: 2222118 bytes. +default 4m19s Normal Started pod/gpu-hold-4 Container started +default 4m19s Normal Created pod/gpu-hold-3 Container created +default 4m17s Normal Killing pod/gpu-hold-1 Stopping container hold +kube-system 4m14s Warning Unhealthy pod/metrics-server-778c4649b4-l4d4j Readiness probe failed: HTTP probe failed with statuscode: 500 +default 3m46s Normal Killing pod/gpu-hold-2 Stopping container hold +default 3m15s Normal Killing pod/gpu-hold-3 Stopping container hold +default 2m44s Normal Killing pod/gpu-hold-4 Stopping container hold +default 2m13s Normal ScalingReplicaSet deployment/vllm-qwen38-27b-xpu Scaled up replica set vllm-qwen38-27b-xpu-799dc85d7c from 0 to 1 +default 2m13s Normal SuccessfulCreate replicaset/vllm-qwen38-27b-xpu-799dc85d7c Created pod: vllm-qwen38-27b-xpu-799dc85d7c-ng77g +default 2m12s Normal Pulling pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Pulling image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" +default 28s Normal Pulled pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Successfully pulled image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" in 1m43.931s (1m43.931s including waiting). Image size: 4143776505 bytes. +default 11s Normal Created pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Container created +default 11s Normal Started pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Container started +default 11s Normal Pulled pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Container image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" already present on machine and can be accessed by the pod +default 10s Warning BackOff pod/vllm-qwen38-27b-xpu-799dc85d7c-ng77g Back-off restarting failed container vllm in pod vllm-qwen38-27b-xpu-799dc85d7c-ng77g_default(6077fde9-5952-450e-b869-3aa47c254b6a) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/gpu-hold.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/gpu-hold.txt new file mode 100644 index 00000000..19ea6376 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/gpu-hold.txt @@ -0,0 +1,4 @@ +gpu-hold-1 renderD128 0000:12:00.0 +gpu-hold-2 renderD130 0000:7c:00.0 +gpu-hold-3 renderD131 0000:96:00.0 +gpu-hold-4 renderD129 0000:48:00.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/live.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/live.log new file mode 100644 index 00000000..177e6bd6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/live.log @@ -0,0 +1,5 @@ +== arm H53 start 00:20:14 (+0s) +== k3d-gpu up 00:20:15 (+1s) +== wait-ready 00:23:06 (+172s) +== cleanup 00:25:19 (+305s) +== done 00:25:22 (+308s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.previous.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.previous.txt new file mode 100644 index 00000000..597ea0e6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.previous.txt @@ -0,0 +1 @@ +--: line 1: /opt/intel/oneapi/setvars.sh: No such file or directory diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt new file mode 100644 index 00000000..597ea0e6 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/log-vllm-qwen38-27b-xpu-799dc85d7c-ng77g.txt @@ -0,0 +1 @@ +--: line 1: /opt/intel/oneapi/setvars.sh: No such file or directory diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pin.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pin.yaml new file mode 100644 index 00000000..96094b70 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pin.yaml @@ -0,0 +1,2 @@ +image: + tag: "v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pods.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pods.txt new file mode 100644 index 00000000..94ce1df0 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/pods.txt @@ -0,0 +1,10 @@ +NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +default vllm-qwen38-27b-xpu-799dc85d7c-ng77g 0/1 CrashLoopBackOff 2 (11s ago) 2m13s 10.42.0.14 k3d-k3dgpu-e-h53-server-0 +kube-system coredns-577d995dff-c89zz 1/1 Running 0 4m48s 10.42.0.2 k3d-k3dgpu-e-h53-server-0 +kube-system helm-install-gateway-api-crd-gfhvl 0/1 Completed 0 4m48s 10.42.0.4 k3d-k3dgpu-e-h53-server-0 +kube-system intel-gpu-plugin-hxvpb 1/1 Running 0 4m48s 10.42.0.5 k3d-k3dgpu-e-h53-server-0 +kube-system local-path-provisioner-6858d854cf-8t662 1/1 Running 0 4m48s 10.42.0.6 k3d-k3dgpu-e-h53-server-0 +kube-system metrics-server-778c4649b4-l4d4j 1/1 Running 0 4m48s 10.42.0.3 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-gc-7d6955597d-v5ns2 1/1 Running 0 4m48s 10.42.0.8 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-master-74b6cfb7df-m4whr 1/1 Running 0 4m48s 10.42.0.9 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-worker-87d64 1/1 Running 0 4m48s 10.42.0.7 k3d-k3dgpu-e-h53-server-0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/reset-mcp.sh b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/reset-mcp.sh new file mode 100755 index 00000000..9730153a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/reset-mcp.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +kill "$(cat "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold/mcp.pid")" 2>/dev/null +for _ in $(seq 50); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null || break; sleep 0.1; done +cd "/home/devcloud/k3dgpu-bench/eab/usecases/banking-assistant/src" && (MCP_HOST=127.0.0.1 MCP_PORT=18600 exec "/home/devcloud/k3dgpu-bench/mafvenv/bin/python" -m mcp_server.server >> "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold/mcp.log" 2>&1 & + echo $! > "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold/mcp.pid") +for _ in $(seq 120); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null && exit 0; sleep 0.5; done +exit 1 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/up.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/up.log new file mode 100644 index 00000000..06d0310e --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-a/hold/up.log @@ -0,0 +1,32 @@ +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators: intel amx +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-e-h53' with image cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-e-h53' +INFO[0000] Created image volume k3d-k3dgpu-e-h53-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-e-h53-tools' +INFO[0001] Creating node 'k3d-k3dgpu-e-h53-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-e-h53' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-e-h53-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-e-h53' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-e-h53) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold/kubeconfig +==> Waiting for nodes Ready... +node/k3d-k3dgpu-e-h53-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +==> intel device plugin auto-deployed by k3s. +==> Verifying accelerators are schedulable... +daemon set "intel-gpu-plugin" successfully rolled out +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators schedulable: intel=4 amx=1. Try: k3d-gpu test diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/b-listing.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/b-listing.txt new file mode 100644 index 00000000..de902c4d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/b-listing.txt @@ -0,0 +1,34 @@ +.gitattributes 1570 1793310278.7959648510 +LICENSE 11544 1793310278.8089651410 +README.md 65012 1793310278.8199653860 +chat_template.jinja 8952 1793310278.8139652520 +config.json 4312 1793310278.8129652300 +crc32.txt 238 1793310278.8299656090 +generation_config.json 202 1793310278.8259655200 +merges.txt 3353259 1793310278.8589662550 +model-00001-of-00018.safetensors 3966730552 1793310507.8016561670 +model-00002-of-00018.safetensors 3043080328 1793310511.8507358520 +model-00003-of-00018.safetensors 2542796952 1793310459.7537105830 +model-00004-of-00018.safetensors 3988973152 1793310571.2509048500 +model-00005-of-00018.safetensors 2099339864 1793310430.8621419970 +model-00006-of-00018.safetensors 3979553696 1793310509.6426923980 +model-00007-of-00018.safetensors 2108759344 1793310436.7122571270 +model-00008-of-00018.safetensors 3979553696 1793310573.9919587940 +model-00009-of-00018.safetensors 2108759344 1793310591.7193076680 +model-00010-of-00018.safetensors 3979553696 1793310721.1158541950 +model-00011-of-00018.safetensors 2108759344 1793310632.3041063770 +model-00012-of-00018.safetensors 3979553696 1793310730.4580380490 +model-00013-of-00018.safetensors 2108759344 1793310660.3256578430 +model-00014-of-00018.safetensors 3979553696 1793310752.6844754650 +model-00015-of-00018.safetensors 2108759344 1793310664.6107421730 +model-00016-of-00018.safetensors 3979564040 1793310758.3635872300 +model-00017-of-00018.safetensors 2108759344 1793310689.8802394770 +model-00018-of-00018.safetensors 3392197344 1793310751.2064463780 +model.safetensors.index.json 112216 1793310660.4686606570 +preprocessor_config.json 390 1793310660.5806628610 +tokenizer.json 12809320 1793310661.7056850020 +tokenizer_config.json 17928 1793310661.8266873830 +video_preprocessor_config.json 385 1793310661.9656901190 +vocab.json 6722759 1793310662.3626979320 +77042094076611b69791a610065f28b7013b8c621795fa86ddccc8bac7d1b9df model.safetensors.index.json +191e0af232104ed8b65258cf3fb2b842e288008baca7633c11b82a1ac7203aab config.json diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/blueprint-commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/blueprint-commit.txt new file mode 100644 index 00000000..2cf3b989 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/blueprint-commit.txt @@ -0,0 +1 @@ +758d4845520ca5456bf732264ef02d92a26d25a6 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/commit.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/commit.txt new file mode 100644 index 00000000..4af4a96a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/commit.txt @@ -0,0 +1 @@ +1f39488be7695dc0d1848347bf1c91f04025bc55 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/deploy.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/deploy.yaml new file mode 100644 index 00000000..864f6f94 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/deploy.yaml @@ -0,0 +1,188 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: vllm-qwen38-27b-xpu-config + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +data: + http_proxy: '' + https_proxy: '' + no_proxy: '' + TORCHINDUCTOR_CACHE_DIR: /tmp + NUMBA_CACHE_DIR: /tmp + HF_HOME: /data + OUTLINES_CACHE_DIR: /tmp/.cache/outlines + VLLM_NO_USAGE_STATS: '1' + DO_NOT_TRACK: '1' + CCL_ALLREDUCE: ring + CCL_PROCESS_LAUNCHER: none + CCL_WORKER_AFFINITY: '' + CCL_WORKER_OFFLOAD: '0' + CCL_ZE_IPC_EXCHANGE: pidfd + HF_ENDPOINT: https://huggingface.co + HF_HUB_DISABLE_XET: '1' + HF_HUB_ENABLE_HF_TRANSFER: '0' + HOME: /tmp + LOGNAME: vllm + ONEAPI_DEVICE_SELECTOR: level_zero:* + SYCL_CACHE_PERSISTENT: '1' + TORCH_LLM_ALLREDUCE: '1' + TRITON_CACHE_DIR: /tmp/.triton + TRITON_HOME: /tmp/.triton + USER: vllm + VLLM_ALLOW_LONG_MAX_MODEL_LEN: '1' + VLLM_ATTENTION_BACKEND: TORCH_SDPA + VLLM_TARGET_DEVICE: xpu + VLLM_USE_V1: '1' + VLLM_WORKER_MULTIPROC_METHOD: spawn + XDG_CACHE_HOME: /tmp/.cache + ZES_ENABLE_SYSMAN: '1' + ZE_ENABLE_PCI_ID_DEVICE_ORDER: '1' + ZE_FLAT_DEVICE_HIERARCHY: FLAT + HF_HUB_OFFLINE: '1' +--- +apiVersion: v1 +kind: Service +metadata: + name: vllm-qwen38-27b-xpu-service + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 8000 + protocol: TCP + name: vllm + selector: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-qwen38-27b-xpu + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: '0.5' + app.kubernetes.io/managed-by: Helm +spec: + selector: + matchLabels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + template: + metadata: + annotations: null + labels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + spec: + securityContext: + fsGroup: 1001 + supplementalGroups: + - 990 + - 44 + containers: + - name: vllm + envFrom: + - configMapRef: + name: vllm-qwen38-27b-xpu-config + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - SYS_NICE + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: false + runAsUser: 0 + seccompProfile: + type: RuntimeDefault + image: vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + imagePullPolicy: IfNotPresent + command: + - bash + - -c + - python3 -m vllm.entrypoints.openai.api_server "$@" + - -- + args: + - --host + - 0.0.0.0 + - --dtype + - float16 + - --block-size + - '64' + - --gpu-memory-utilization + - '0.9' + - --enforce-eager + - --no-enable-prefix-caching + - --trust-remote-code + - --disable-sliding-window + - --enable-auto-tool-choice + - --tool-call-parser + - qwen3_xml + - --tensor-parallel-size + - '4' + - --model + - /data/Qwen3.8-27B + - --served-model-name + - Qwen/Qwen3.6-27B + - --port + - '8000' + volumeMounts: + - mountPath: /data + name: model-volume + - mountPath: /dev/shm + name: shm + - mountPath: /tmp + name: tmp + - mountPath: /root/.cache/vllm + name: vllm-cache + ports: + - name: http + containerPort: 8000 + protocol: TCP + readinessProbe: + failureThreshold: 200 + httpGet: + path: /health + port: http + initialDelaySeconds: 10 + periodSeconds: 5 + resources: + limits: + gpu.intel.com/xe: '4' + volumes: + - name: model-volume + hostPath: + path: /var/apb-ro + type: Directory + - name: shm + emptyDir: + medium: Memory + sizeLimit: 1Gi + - name: tmp + emptyDir: {} + - name: vllm-cache + emptyDir: {} + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: ei-inference-eligible + operator: In + values: + - 'true' diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/describe-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/describe-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt new file mode 100644 index 00000000..1743b33f --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/describe-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt @@ -0,0 +1,119 @@ +Name: vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz +Namespace: default +Priority: 0 +Service Account: default +Node: k3d-k3dgpu-e-h53-server-0/172.19.0.2 +Start Time: Fri, 30 Oct 2026 00:28:27 +0000 +Labels: app.kubernetes.io/instance=vllm-qwen38-27b-xpu + app.kubernetes.io/name=vllm + pod-template-hash=69c59cbbdc +Annotations: +Status: Running +IP: 10.42.0.14 +IPs: + IP: 10.42.0.14 +Controlled By: ReplicaSet/vllm-qwen38-27b-xpu-69c59cbbdc +Containers: + vllm: + Container ID: containerd://3eae540505cef3060d89c39dbcda364b817f1c88832c85389570dc07ce3514c5 + Image: vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + Image ID: docker.io/vllm/vllm-openai-xpu@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3 + Port: 8000/TCP (http) + Host Port: 0/TCP (http) + SeccompProfile: RuntimeDefault + Command: + bash + -c + python3 -m vllm.entrypoints.openai.api_server "$@" + -- + Args: + --host + 0.0.0.0 + --dtype + float16 + --block-size + 64 + --gpu-memory-utilization + 0.9 + --enforce-eager + --no-enable-prefix-caching + --trust-remote-code + --disable-sliding-window + --enable-auto-tool-choice + --tool-call-parser + qwen3_xml + --tensor-parallel-size + 4 + --model + /data/Qwen3.8-27B + --served-model-name + Qwen/Qwen3.6-27B + --port + 8000 + State: Running + Started: Fri, 30 Oct 2026 00:36:01 +0000 + Last State: Terminated + Reason: Error + Exit Code: 1 + Started: Fri, 30 Oct 2026 00:33:56 +0000 + Finished: Fri, 30 Oct 2026 00:35:32 +0000 + Ready: False + Restart Count: 3 + Limits: + gpu.intel.com/xe: 4 + Requests: + gpu.intel.com/xe: 4 + Readiness: http-get http://:http/health delay=10s timeout=1s period=5s successThreshold=1 failureThreshold=200 + Environment Variables from: + vllm-qwen38-27b-xpu-config ConfigMap Optional: false + Environment: + Mounts: + /data from model-volume (rw) + /dev/shm from shm (rw) + /root/.cache/vllm from vllm-cache (rw) + /tmp from tmp (rw) + /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hq2fx (ro) +Conditions: + Type Status + PodReadyToStartContainers True + Initialized True + Ready False + ContainersReady False + PodScheduled True +Volumes: + model-volume: + Type: HostPath (bare host directory volume) + Path: /var/apb-ro + HostPathType: Directory + shm: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: Memory + SizeLimit: 1Gi + tmp: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: + vllm-cache: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: + kube-api-access-hq2fx: + Type: Projected (a volume that contains injected data from multiple sources) + TokenExpirationSeconds: 3607 + ConfigMapName: kube-root-ca.crt + Optional: false + DownwardAPI: true +QoS Class: BestEffort +Node-Selectors: +Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s + node.kubernetes.io/unreachable:NoExecute op=Exists for 300s +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 7m37s default-scheduler Successfully assigned default/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz to k3d-k3dgpu-e-h53-server-0 + Normal Pulling 7m37s kubelet spec.containers{vllm}: Pulling image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" + Normal Pulled 5m52s kubelet spec.containers{vllm}: Successfully pulled image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" in 1m44.709s (1m44.709s including waiting). Image size: 4143776505 bytes. + Warning Unhealthy 37s (x58 over 5m41s) kubelet spec.containers{vllm}: Readiness probe failed: Get "http://10.42.0.14:8000/health": dial tcp 10.42.0.14:8000: connect: connection refused + Normal Created 3s (x4 over 5m52s) kubelet spec.containers{vllm}: Container created + Normal Started 3s (x4 over 5m52s) kubelet spec.containers{vllm}: Container started + Normal Pulled 3s (x3 over 3m57s) kubelet spec.containers{vllm}: Container image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" already present on machine and can be accessed by the pod diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/devdri-diag.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/devdri-diag.txt new file mode 100644 index 00000000..02568fcc --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/devdri-diag.txt @@ -0,0 +1,72 @@ +2026-10-30T00:36:13Z +## host: ls -la /dev/dri /dev/dri/by-path +/dev/dri: +total 0 +drwxr-xr-x 3 root root 240 Sep 15 22:48 . +drwxr-xr-x 19 root root 6480 Oct 29 18:47 .. +drwxr-xr-x 2 root root 240 Sep 15 22:48 by-path +crw-rw----+ 1 root video 226, 0 Sep 16 00:04 card0 +crw-rw----+ 1 root video 226, 1 Sep 16 00:04 card1 +crw-rw----+ 1 root video 226, 2 Sep 16 00:04 card2 +crw-rw----+ 1 root video 226, 3 Sep 16 00:04 card3 +crw-rw----+ 1 root video 226, 4 Sep 16 00:04 card4 +crw-rw----+ 1 root render 226, 128 Sep 16 00:04 renderD128 +crw-rw----+ 1 root render 226, 129 Sep 16 00:04 renderD129 +crw-rw----+ 1 root render 226, 130 Sep 16 00:04 renderD130 +crw-rw----+ 1 root render 226, 131 Sep 16 00:04 renderD131 + +/dev/dri/by-path: +total 0 +drwxr-xr-x 2 root root 240 Sep 15 22:48 . +drwxr-xr-x 3 root root 240 Sep 15 22:48 .. +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:12:00.0-card -> ../card1 +lrwxrwxrwx+1 root root 13 Sep 15 22:48 pci-0000:12:00.0-render -> ../renderD128 +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:2d:00.0-card -> ../card0 +lrwxrwxrwx+1 root root 8 Sep 15 22:47 pci-0000:2d:00.0-platform-simple-framebuffer.0-card -> ../card0 +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:48:00.0-card -> ../card2 +lrwxrwxrwx+1 root root 13 Sep 15 22:48 pci-0000:48:00.0-render -> ../renderD129 +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:7c:00.0-card -> ../card3 +lrwxrwxrwx+1 root root 13 Sep 15 22:48 pci-0000:7c:00.0-render -> ../renderD130 +lrwxrwxrwx+1 root root 8 Sep 15 22:48 pci-0000:96:00.0-card -> ../card4 +lrwxrwxrwx+1 root root 13 Sep 15 22:48 pci-0000:96:00.0-render -> ../renderD131 +## node k3d-k3dgpu-e-h53-server-0: ls -la /dev/dri /dev/dri/by-path +/dev/dri: +total 0 +drwxr-xr-x 2 root root 220 Oct 30 00:25 . +drwxr-xr-x 14 root root 6300 Oct 30 00:25 .. +crw-rw---- 1 root video 226, 0 Oct 30 00:25 card0 +crw-rw---- 1 root video 226, 1 Oct 30 00:25 card1 +crw-rw---- 1 root video 226, 2 Oct 30 00:25 card2 +crw-rw---- 1 root video 226, 3 Oct 30 00:25 card3 +crw-rw---- 1 root video 226, 4 Oct 30 00:25 card4 +crw-rw---- 1 root 990 226, 128 Oct 30 00:25 renderD128 +crw-rw---- 1 root 990 226, 129 Oct 30 00:25 renderD129 +crw-rw---- 1 root 990 226, 130 Oct 30 00:25 renderD130 +crw-rw---- 1 root 990 226, 131 Oct 30 00:25 renderD131 +ls: cannot access '/dev/dri/by-path': No such file or directory +## pod pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz: ls -la /dev/dri /dev/dri/by-path +ls: cannot access '/dev/dri/by-path': No such file or directory +/dev/dri: +total 0 +drwxr-xr-x 2 root root 200 Oct 30 00:36 . +drwxr-xr-x 6 root root 380 Oct 30 00:36 .. +crw-rw---- 1 root root 226, 1 Oct 30 00:36 card1 +crw-rw---- 1 root root 226, 2 Oct 30 00:36 card2 +crw-rw---- 1 root root 226, 3 Oct 30 00:36 card3 +crw-rw---- 1 root root 226, 4 Oct 30 00:36 card4 +crw-rw---- 1 root root 226, 128 Oct 30 00:36 renderD128 +crw-rw---- 1 root root 226, 129 Oct 30 00:36 renderD129 +crw-rw---- 1 root root 226, 130 Oct 30 00:36 renderD130 +crw-rw---- 1 root root 226, 131 Oct 30 00:36 renderD131 +command terminated with exit code 2 +## pod: /dev/dri strings in the image oneCCL libraries +/opt/venv/lib/libccl.so /opt/venv/lib/libccl.so + 1 /dev/dri/by-path/ +/opt/venv/lib/libccl.so.1 /opt/venv/lib/libccl.so.1 + 1 /dev/dri/by-path/ +/opt/venv/lib/libccl.so.1.0 /opt/venv/lib/libccl.so.1.0 + 1 /dev/dri/by-path/ +/opt/venv/lib/libccl.so.2 /opt/venv/lib/libccl.so.2 +/opt/venv/lib/libccl.so.2.0 /opt/venv/lib/libccl.so.2.0 +## intel-gpu-plugin daemonset args and volumes +intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33 [{"hostPath":{"path":"/dev/dri","type":"DirectoryOrCreate"},"name":"devfs"},{"hostPath":{"path":"/sys/class/drm","type":""},"name":"sysfsdrm"},{"hostPath":{"path":"/var/lib/kubelet/device-plugins","type":""},"name":"kubeletsockets"},{"hostPath":{"path":"/var/run/cdi","type":"DirectoryOrCreate"},"name":"cdipath"}] diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/eat.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/eat.yaml new file mode 100644 index 00000000..7c2d0407 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/eat.yaml @@ -0,0 +1,301 @@ +--- +# Source: vllm/templates/secret.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: Secret +metadata: + name: vllm-qwen38-27b-xpu-secret + labels: + app: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +type: Opaque +data: +--- +# Source: vllm/templates/configmap.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: ConfigMap +metadata: + name: vllm-qwen38-27b-xpu-config + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +data: + http_proxy: "" + https_proxy: "" + no_proxy: "" + TORCHINDUCTOR_CACHE_DIR: "/tmp" + NUMBA_CACHE_DIR: "/tmp" + HF_HOME: "/data" + OUTLINES_CACHE_DIR: "/tmp/.cache/outlines" + VLLM_NO_USAGE_STATS: "1" + DO_NOT_TRACK: "1" + CCL_ALLREDUCE: "ring" + CCL_PROCESS_LAUNCHER: "none" + CCL_WORKER_AFFINITY: "" + CCL_WORKER_OFFLOAD: "0" + CCL_ZE_IPC_EXCHANGE: "pidfd" + DO_NOT_TRACK: "1" + HF_ENDPOINT: "https://huggingface.co" + HF_HUB_DISABLE_XET: "1" + HF_HUB_ENABLE_HF_TRANSFER: "0" + HOME: "/tmp" + LOGNAME: "vllm" + ONEAPI_DEVICE_SELECTOR: "level_zero:*" + SYCL_CACHE_PERSISTENT: "1" + TORCH_LLM_ALLREDUCE: "1" + TRITON_CACHE_DIR: "/tmp/.triton" + TRITON_HOME: "/tmp/.triton" + USER: "vllm" + VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1" + VLLM_ATTENTION_BACKEND: "TORCH_SDPA" + VLLM_NO_USAGE_STATS: "1" + VLLM_TARGET_DEVICE: "xpu" + VLLM_USE_V1: "1" + VLLM_WORKER_MULTIPROC_METHOD: "spawn" + XDG_CACHE_HOME: "/tmp/.cache" + ZES_ENABLE_SYSMAN: "1" + ZE_ENABLE_PCI_ID_DEVICE_ORDER: "1" + ZE_FLAT_DEVICE_HIERARCHY: "FLAT" +--- +# Source: vllm/templates/pvc.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: vllm-qwen38-27b-xpu-pvc + namespace: default +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 250Gi # Adjust this according to model storage size +--- +# Source: vllm/templates/service.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: v1 +kind: Service +metadata: + name: vllm-qwen38-27b-xpu-service + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 8000 + protocol: TCP + name: vllm + selector: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu +--- +# Source: vllm/templates/deployment.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-qwen38-27b-xpu + labels: + helm.sh/chart: vllm-0-latest + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + app.kubernetes.io/version: "0.5" + app.kubernetes.io/managed-by: Helm +spec: + selector: + matchLabels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + template: + metadata: + annotations: + labels: + app.kubernetes.io/name: vllm + app.kubernetes.io/instance: vllm-qwen38-27b-xpu + spec: + # + # serviceAccountName: default + securityContext: + fsGroup: 1001 + supplementalGroups: + - 990 + - 44 + containers: + - name: vllm + envFrom: + - configMapRef: + name: vllm-qwen38-27b-xpu-config + - secretRef: + name: vllm-qwen38-27b-xpu-secret + securityContext: + allowPrivilegeEscalation: false + capabilities: + add: + - SYS_NICE + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: false + runAsUser: 0 + seccompProfile: + type: RuntimeDefault + image: "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" + imagePullPolicy: IfNotPresent + command: + - bash + - -c + - source /opt/intel/oneapi/setvars.sh --force && export LD_LIBRARY_PATH=/host-gpu-libs:${LD_LIBRARY_PATH} + && python3 -m vllm.entrypoints.openai.api_server "$@" + - -- + # command: + # - /bin/bash + # - -c + # - | + # python3 -m vllm.entrypoints.openai.api_server --dtype float16 --model Qwen/Qwen3.6-27B --port 8000 --tensor-parallel-size 4 --block-size 64 --max-model-len 32768 --disable-log-requests + args: + - "--host" + - "0.0.0.0" + - "--dtype" + - "float16" + - "--block-size" + - "64" + - "--gpu-memory-utilization" + - "0.9" + - "--enforce-eager" + - "--no-enable-prefix-caching" + - "--trust-remote-code" + - "--disable-sliding-window" + - "--enable-auto-tool-choice" + - "--tool-call-parser" + - "qwen3_xml" + - "--tensor-parallel-size" + - "4" + - "--model" + - "Qwen/Qwen3.6-27B" + - --served-model-name + - "Qwen/Qwen3.6-27B" + - "--port" + - "8000" + volumeMounts: + - mountPath: /data + name: model-volume + - mountPath: /dev/shm + name: shm + - mountPath: /tmp + name: tmp + - mountPath: /root/.cache/vllm + name: vllm-cache + - mountPath: /host-gpu-libs + name: host-gpu-libs + readOnly: true + - mountPath: /usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1 + name: host-gpu-libs + readOnly: true + subPath: libze_intel_gpu.so.1 + - mountPath: /usr/lib/x86_64-linux-gnu/libze_loader.so.1 + name: host-gpu-libs + readOnly: true + subPath: libze_loader.so.1 + - mountPath: /etc/OpenCL/vendors + name: host-opencl-vendors + readOnly: true + - mountPath: /usr/lib/x86_64-linux-gnu/intel-opencl + name: host-intel-opencl + readOnly: true + ports: + - name: http + containerPort: 8000 + protocol: TCP + readinessProbe: + failureThreshold: 98 + httpGet: + path: /health + port: http + initialDelaySeconds: 120 + periodSeconds: 30 + resources: + limits: + "gpu.intel.com/xe": "4" + + volumes: + - name: model-volume + persistentVolumeClaim: + claimName: vllm-qwen38-27b-xpu-pvc + - name: shm + emptyDir: + medium: Memory + sizeLimit: 1Gi + - name: tmp + emptyDir: {} + - name: vllm-cache + emptyDir: {} + - hostPath: + path: /usr/lib/x86_64-linux-gnu + type: Directory + name: host-gpu-libs + - hostPath: + path: /etc/OpenCL/vendors + type: Directory + name: host-opencl-vendors + - hostPath: + path: /usr/lib/x86_64-linux-gnu/intel-opencl + type: Directory + name: host-intel-opencl + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: ei-inference-eligible + operator: In + values: + - "true" + +--- +# Source: vllm/templates/apisixroutes.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +--- +# Source: vllm/templates/chat-template-configmap.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +--- +# Source: vllm/templates/ingress.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# Please edit the object below. Lines beginning with a '#' will be ignored, +# and an empty file will abort the edit. If an error occurs while saving this file will be +# reopened with the relevant failures. +--- +# Source: vllm/templates/ingress_eks.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +--- +# Source: vllm/templates/route.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +--- +# Source: vllm/templates/servicemonitor.yaml +# Copyright (C) 2025-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/events.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/events.txt new file mode 100644 index 00000000..a98c3218 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/events.txt @@ -0,0 +1,164 @@ +NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE +node-feature-discovery 10m Normal Scheduled pod/nfd-master-74b6cfb7df-6wq4q Successfully assigned node-feature-discovery/nfd-master-74b6cfb7df-6wq4q to k3d-k3dgpu-e-h53-server-0 +default 10m Normal Starting node/k3d-k3dgpu-e-h53-server-0 +default 9m45s Normal Scheduled pod/gpu-hold-1 Successfully assigned default/gpu-hold-1 to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/intel-gpu-plugin-sfv7x Successfully assigned kube-system/intel-gpu-plugin-sfv7x to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 10m Normal Scheduled pod/nfd-worker-j8s9j Successfully assigned node-feature-discovery/nfd-worker-j8s9j to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/metrics-server-778c4649b4-gp9l4 Successfully assigned kube-system/metrics-server-778c4649b4-gp9l4 to k3d-k3dgpu-e-h53-server-0 +default 9m45s Normal Scheduled pod/gpu-hold-2 Successfully assigned default/gpu-hold-2 to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/helm-install-gateway-api-crd-n4z4l Successfully assigned kube-system/helm-install-gateway-api-crd-n4z4l to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/coredns-577d995dff-bsdh4 Successfully assigned kube-system/coredns-577d995dff-bsdh4 to k3d-k3dgpu-e-h53-server-0 +default 9m45s Normal Scheduled pod/gpu-hold-3 Successfully assigned default/gpu-hold-3 to k3d-k3dgpu-e-h53-server-0 +kube-system 10m Normal Scheduled pod/local-path-provisioner-6858d854cf-wx2r9 Successfully assigned kube-system/local-path-provisioner-6858d854cf-wx2r9 to k3d-k3dgpu-e-h53-server-0 +default 9m45s Normal Scheduled pod/gpu-hold-4 Successfully assigned default/gpu-hold-4 to k3d-k3dgpu-e-h53-server-0 +default 7m37s Normal Scheduled pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Successfully assigned default/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz to k3d-k3dgpu-e-h53-server-0 +node-feature-discovery 10m Normal Scheduled pod/nfd-gc-7d6955597d-28trj Successfully assigned node-feature-discovery/nfd-gc-7d6955597d-28trj to k3d-k3dgpu-e-h53-server-0 +default 10m Normal NodeAllocatableEnforced node/k3d-k3dgpu-e-h53-server-0 Updated Node Allocatable limit across pods +default 10m Normal NodeHasSufficientPID node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasSufficientPID +default 10m Normal NodeHasNoDiskPressure node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasNoDiskPressure +default 10m Normal CertificateExpirationOK node/k3d-k3dgpu-e-h53-server-0 Node and Certificate Authority certificates managed by k3s are OK +default 10m Normal NodeHasSufficientMemory node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeHasSufficientMemory +default 10m Warning InvalidDiskCapacity node/k3d-k3dgpu-e-h53-server-0 invalid capacity 0 on image filesystem +default 10m Normal Starting node/k3d-k3dgpu-e-h53-server-0 Starting kubelet. +default 10m Normal NodeReady node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 status is now: NodeReady +kube-system 10m Normal DeletingManifest addon/intel-gpu-dra Deleting manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-dra.yaml" +kube-system 10m Normal ApplyingManifest addon/ccm Applying manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-gpu-wsl-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-wsl-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/intel-gpu-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-gpu-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/intel-dsa-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-dsa-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-dsa-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-dsa-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/gaudi-device-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/gaudi-device-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/gaudi-device-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/gaudi-device-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/gateway-api-crd Applied manifest at "/var/lib/rancher/k3s/server/manifests/gateway-api-crd.yaml" +kube-system 10m Normal ApplyingManifest addon/gateway-api-crd Applying manifest at "/var/lib/rancher/k3s/server/manifests/gateway-api-crd.yaml" +kube-system 10m Normal AppliedManifest addon/coredns Applied manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml" +kube-system 10m Normal ApplyingManifest addon/coredns Applying manifest at "/var/lib/rancher/k3s/server/manifests/coredns.yaml" +kube-system 10m Normal AppliedManifest addon/ccm Applied manifest at "/var/lib/rancher/k3s/server/manifests/ccm.yaml" +kube-system 10m Normal ApplyJob helmchart/gateway-api-crd Applying HelmChart from https://%{KUBERNETES_API}%/static/charts/gateway-api-crd-1.6.103.tgz using Job kube-system/helm-install-gateway-api-crd +kube-system 10m Normal ApplyingManifest addon/intel-iaa-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-iaa-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/metrics-apiservice Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" +kube-system 10m Normal ApplyingManifest addon/metrics-apiservice Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml" +kube-system 10m Normal AppliedManifest addon/local-storage Applied manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml" +kube-system 10m Normal ApplyingManifest addon/local-storage Applying manifest at "/var/lib/rancher/k3s/server/manifests/local-storage.yaml" +kube-system 10m Normal ApplyingManifest addon/node-feature-discovery Applying manifest at "/var/lib/rancher/k3s/server/manifests/node-feature-discovery.yaml" +kube-system 10m Normal AppliedManifest addon/intel-qat-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-qat-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/metrics-server-service Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" +kube-system 10m Normal ApplyingManifest addon/aggregated-metrics-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" +kube-system 10m Normal AppliedManifest addon/aggregated-metrics-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml" +kube-system 10m Normal ApplyingManifest addon/auth-delegator Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" +kube-system 10m Normal AppliedManifest addon/auth-delegator Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml" +kube-system 10m Normal ApplyingManifest addon/auth-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" +kube-system 10m Normal AppliedManifest addon/auth-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-qat-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-qat-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/intel-npu-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-npu-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/intel-npu-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/intel-npu-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/intel-iaa-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-iaa-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/resource-reader Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" +kube-system 10m Normal AppliedManifest addon/intel-gpu-wsl-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/intel-gpu-wsl-plugin.yaml" +kube-system 10m Normal ApplyingManifest addon/metrics-server-deployment Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" +kube-system 10m Normal AppliedManifest addon/metrics-server-deployment Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml" +kube-system 10m Normal ApplyingManifest addon/metrics-server-service Applying manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml" +kube-system 10m Normal AppliedManifest addon/resource-reader Applied manifest at "/var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml" +default 10m Normal NodePasswordValidationComplete node/k3d-k3dgpu-e-h53-server-0 Deferred node password secret validation complete +default 10m Normal Synced node/k3d-k3dgpu-e-h53-server-0 Node synced successfully +kube-system 10m Normal ApplyingManifest addon/runtimes Applying manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml" +kube-system 10m Normal AppliedManifest addon/runtimes Applied manifest at "/var/lib/rancher/k3s/server/manifests/runtimes.yaml" +default 10m Normal RegisteredNode node/k3d-k3dgpu-e-h53-server-0 Node k3d-k3dgpu-e-h53-server-0 event: Registered Node k3d-k3dgpu-e-h53-server-0 in Controller +kube-system 10m Normal ApplyingManifest addon/rolebindings Applying manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml" +kube-system 10m Normal ResumeJob helmchart/gateway-api-crd Resumed synced Job kube-system/helm-install-gateway-api-crd +kube-system 10m Normal AppliedManifest addon/nvidia-device-plugin Applied manifest at "/var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml" +kube-system 10m Normal AppliedManifest addon/rolebindings Applied manifest at "/var/lib/rancher/k3s/server/manifests/rolebindings.yaml" +node-feature-discovery 10m Normal SuccessfulCreate daemonset/nfd-worker Created pod: nfd-worker-j8s9j +kube-system 10m Normal DeletingManifest addon/traefik Deleting manifest at "/var/lib/rancher/k3s/server/manifests/traefik.yaml" +kube-system 10m Normal Resumed job/helm-install-gateway-api-crd Job resumed +kube-system 10m Normal ApplyingManifest addon/nvidia-device-plugin Applying manifest at "/var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml" +kube-system 10m Normal DeletingManifest addon/nri-balloons Deleting manifest at "/var/lib/rancher/k3s/server/manifests/nri-balloons.yaml" +kube-system 10m Normal AppliedManifest addon/node-feature-discovery Applied manifest at "/var/lib/rancher/k3s/server/manifests/node-feature-discovery.yaml" +kube-system 10m Normal SuccessfulCreate job/helm-install-gateway-api-crd Created pod: helm-install-gateway-api-crd-n4z4l +kube-system 10m Normal Suspended job/helm-install-gateway-api-crd Job suspended +kube-system 10m Warning FailedCreatePodSandBox pod/metrics-server-778c4649b4-gp9l4 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4f8399710121c9a074b4cd33924be84f0033a52cade1a44b25cd75f00bcc4e26": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 10m Normal SuccessfulCreate replicaset/local-path-provisioner-6858d854cf Created pod: local-path-provisioner-6858d854cf-wx2r9 +kube-system 10m Normal SuccessfulCreate replicaset/metrics-server-778c4649b4 Created pod: metrics-server-778c4649b4-gp9l4 +node-feature-discovery 10m Normal ScalingReplicaSet deployment/nfd-gc Scaled up replica set nfd-gc-7d6955597d from 0 to 1 +kube-system 10m Warning FailedMount pod/helm-install-gateway-api-crd-n4z4l MountVolume.SetUp failed for volume "kube-api-access-v8jfs" : configmap "kube-root-ca.crt" not found +kube-system 10m Normal ScalingReplicaSet deployment/coredns Scaled up replica set coredns-577d995dff from 0 to 1 +node-feature-discovery 10m Warning FailedCreatePodSandBox pod/nfd-worker-j8s9j Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d0eac7f436aea07971f0b66a8a1c21ad1d05b90b6c25d391bf2847d19714265f": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 10m Normal SuccessfulCreate replicaset/nfd-master-74b6cfb7df Created pod: nfd-master-74b6cfb7df-6wq4q +kube-system 10m Normal SuccessfulCreate replicaset/coredns-577d995dff Created pod: coredns-577d995dff-bsdh4 +kube-system 10m Normal ScalingReplicaSet deployment/local-path-provisioner Scaled up replica set local-path-provisioner-6858d854cf from 0 to 1 +node-feature-discovery 10m Normal ScalingReplicaSet deployment/nfd-master Scaled up replica set nfd-master-74b6cfb7df from 0 to 1 +kube-system 10m Normal ScalingReplicaSet deployment/metrics-server Scaled up replica set metrics-server-778c4649b4 from 0 to 1 +kube-system 10m Warning FailedCreatePodSandBox pod/intel-gpu-plugin-sfv7x Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2285fe50dcd6ea62c3c74cdf96ce93d647de8dd6c94c5c46e141c8e6fb19a3fb": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 10m Warning FailedCreatePodSandBox pod/nfd-gc-7d6955597d-28trj Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5cbcac78bd49b0c5bc76d3f27c874ec598a7492981909c98dcaf26ab0feabe20": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 10m Normal SuccessfulCreate daemonset/intel-gpu-plugin Created pod: intel-gpu-plugin-sfv7x +node-feature-discovery 10m Warning FailedCreatePodSandBox pod/nfd-master-74b6cfb7df-6wq4q Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d4fde5d6750465fb7f87ea30f039d3fe65bfb08120606ad0530a0d2097fc62ad": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 10m Warning FailedCreatePodSandBox pod/local-path-provisioner-6858d854cf-wx2r9 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "705ec65010d1d8a6ffee20b2f8ea22d6931b5f638733c6a363d061b0023c0bbf": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 10m Normal SuccessfulCreate replicaset/nfd-gc-7d6955597d Created pod: nfd-gc-7d6955597d-28trj +kube-system 10m Warning FailedCreatePodSandBox pod/coredns-577d995dff-bsdh4 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "36f697d6bcaab61deb47c6276b332ff1d643e13a5da41df6027d362fe6b88f7a": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +kube-system 10m Warning FailedCreatePodSandBox pod/helm-install-gateway-api-crd-n4z4l Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2cd13b2cc3035a6c20d4040c80975ad185630ea3588c7d4737923e631de4f1cb": plugin type="flannel" failed (add): failed to load flannel 'subnet.env' file: open /run/flannel/subnet.env: no such file or directory. Check the flannel pod log for this node. +node-feature-discovery 9m59s Normal Pulling pod/nfd-master-74b6cfb7df-6wq4q Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +kube-system 9m59s Normal Pulling pod/coredns-577d995dff-bsdh4 Pulling image "rancher/mirrored-coredns-coredns:1.14.7" +kube-system 9m59s Normal Pulling pod/local-path-provisioner-6858d854cf-wx2r9 Pulling image "rancher/local-path-provisioner:v0.0.37" +kube-system 9m58s Normal Pulling pod/metrics-server-778c4649b4-gp9l4 Pulling image "rancher/mirrored-metrics-server:v0.9.0" +node-feature-discovery 9m58s Normal Pulling pod/nfd-worker-j8s9j Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +node-feature-discovery 9m58s Normal Pulling pod/nfd-gc-7d6955597d-28trj Pulling image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" +kube-system 9m57s Normal Pulling pod/intel-gpu-plugin-sfv7x Pulling image "intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33" +kube-system 9m57s Normal Pulling pod/helm-install-gateway-api-crd-n4z4l Pulling image "rancher/klipper-helm:v0.13.3-build20260727" +kube-system 9m56s Normal Pulled pod/coredns-577d995dff-bsdh4 Successfully pulled image "rancher/mirrored-coredns-coredns:1.14.7" in 2.64s (2.64s including waiting). Image size: 24261571 bytes. +kube-system 9m56s Normal Pulled pod/local-path-provisioner-6858d854cf-wx2r9 Successfully pulled image "rancher/local-path-provisioner:v0.0.37" in 2.636s (2.636s including waiting). Image size: 15066012 bytes. +kube-system 9m56s Normal Created pod/local-path-provisioner-6858d854cf-wx2r9 Container created +kube-system 9m56s Normal Started pod/local-path-provisioner-6858d854cf-wx2r9 Container started +kube-system 9m56s Normal Created pod/coredns-577d995dff-bsdh4 Container created +kube-system 9m56s Normal Started pod/coredns-577d995dff-bsdh4 Container started +kube-system 9m56s Warning Unhealthy pod/coredns-577d995dff-bsdh4 Readiness probe failed: Get "http://10.42.0.2:8181/ready": dial tcp 10.42.0.2:8181: connect: connection refused +kube-system 9m55s Warning Unhealthy pod/metrics-server-778c4649b4-gp9l4 Readiness probe failed: Get "https://10.42.0.5:10250/readyz": dial tcp 10.42.0.5:10250: connect: connection refused +node-feature-discovery 9m55s Normal Started pod/nfd-worker-j8s9j Container started +kube-system 9m55s Normal Created pod/intel-gpu-plugin-sfv7x Container created +kube-system 9m55s Normal Pulled pod/metrics-server-778c4649b4-gp9l4 Successfully pulled image "rancher/mirrored-metrics-server:v0.9.0" in 2.962s (2.962s including waiting). Image size: 23258428 bytes. +kube-system 9m55s Normal Created pod/metrics-server-778c4649b4-gp9l4 Container created +kube-system 9m55s Normal Started pod/metrics-server-778c4649b4-gp9l4 Container started +kube-system 9m55s Normal Pulled pod/intel-gpu-plugin-sfv7x Successfully pulled image "intel/intel-gpu-plugin:0.37.1@sha256:004641b2547d58f1eadce58473fa160ad71377de1eb588cf1f82832387bf3e33" in 2.075s (2.075s including waiting). Image size: 5970136 bytes. +node-feature-discovery 9m55s Normal Started pod/nfd-master-74b6cfb7df-6wq4q Container started +node-feature-discovery 9m55s Warning Unhealthy pod/nfd-master-74b6cfb7df-6wq4q Readiness probe failed: Get "http://10.42.0.4:8080/healthz": dial tcp 10.42.0.4:8080: connect: connection refused +node-feature-discovery 9m55s Normal Pulled pod/nfd-gc-7d6955597d-28trj Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 2.234s (2.234s including waiting). Image size: 85412382 bytes. +node-feature-discovery 9m55s Normal Created pod/nfd-gc-7d6955597d-28trj Container created +node-feature-discovery 9m55s Normal Started pod/nfd-gc-7d6955597d-28trj Container started +node-feature-discovery 9m55s Normal Pulled pod/nfd-master-74b6cfb7df-6wq4q Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 3.225s (3.225s including waiting). Image size: 85412382 bytes. +kube-system 9m55s Normal Started pod/intel-gpu-plugin-sfv7x Container started +node-feature-discovery 9m55s Normal Created pod/nfd-worker-j8s9j Container created +node-feature-discovery 9m55s Normal Pulled pod/nfd-worker-j8s9j Successfully pulled image "registry.k8s.io/nfd/node-feature-discovery:v0.19.0@sha256:2fa1c99ad09bdf2c8ad97706a4ad2fd548c84d5ecd70ba32a6152c667b96c4d2" in 2.229s (2.229s including waiting). Image size: 85412382 bytes. +node-feature-discovery 9m55s Normal Created pod/nfd-master-74b6cfb7df-6wq4q Container created +kube-system 9m53s Normal Pulled pod/helm-install-gateway-api-crd-n4z4l Successfully pulled image "rancher/klipper-helm:v0.13.3-build20260727" in 3.609s (3.609s including waiting). Image size: 74172006 bytes. +kube-system 9m53s Normal Created pod/helm-install-gateway-api-crd-n4z4l Container created +kube-system 9m53s Normal Started pod/helm-install-gateway-api-crd-n4z4l Container started +kube-system 9m50s Normal Completed job/helm-install-gateway-api-crd Job completed +default 9m45s Normal Pulling pod/gpu-hold-1 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 9m45s Normal Pulling pod/gpu-hold-3 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 9m45s Normal Pulling pod/gpu-hold-2 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 9m45s Normal Pulling pod/gpu-hold-4 Pulling image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" +default 9m43s Normal Pulled pod/gpu-hold-3 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.345s (1.345s including waiting). Image size: 2222118 bytes. +default 9m43s Normal Started pod/gpu-hold-1 Container started +default 9m43s Normal Created pod/gpu-hold-3 Container created +default 9m43s Normal Pulled pod/gpu-hold-4 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.274s (1.274s including waiting). Image size: 2222118 bytes. +default 9m43s Normal Created pod/gpu-hold-4 Container created +default 9m43s Normal Started pod/gpu-hold-4 Container started +default 9m43s Normal Pulled pod/gpu-hold-1 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.502s (1.502s including waiting). Image size: 2222118 bytes. +default 9m43s Normal Started pod/gpu-hold-3 Container started +default 9m43s Normal Created pod/gpu-hold-1 Container created +default 9m43s Normal Started pod/gpu-hold-2 Container started +default 9m43s Normal Created pod/gpu-hold-2 Container created +default 9m43s Normal Pulled pod/gpu-hold-2 Successfully pulled image "busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec" in 1.413s (1.413s including waiting). Image size: 2222118 bytes. +default 9m42s Normal Killing pod/gpu-hold-1 Stopping container hold +kube-system 9m41s Warning Unhealthy pod/metrics-server-778c4649b4-gp9l4 Readiness probe failed: HTTP probe failed with statuscode: 500 +default 9m10s Normal Killing pod/gpu-hold-2 Stopping container hold +default 8m39s Normal Killing pod/gpu-hold-3 Stopping container hold +default 8m8s Normal Killing pod/gpu-hold-4 Stopping container hold +default 7m37s Normal Pulling pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Pulling image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" +default 7m37s Normal SuccessfulCreate replicaset/vllm-qwen38-27b-xpu-69c59cbbdc Created pod: vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz +default 7m37s Normal ScalingReplicaSet deployment/vllm-qwen38-27b-xpu Scaled up replica set vllm-qwen38-27b-xpu-69c59cbbdc from 0 to 1 +default 5m52s Normal Pulled pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Successfully pulled image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" in 1m44.709s (1m44.709s including waiting). Image size: 4143776505 bytes. +default 37s Warning Unhealthy pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Readiness probe failed: Get "http://10.42.0.14:8000/health": dial tcp 10.42.0.14:8000: connect: connection refused +default 3s Normal Created pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Container created +default 3s Normal Started pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Container started +default 3s Normal Pulled pod/vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz Container image "vllm/vllm-openai-xpu:v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" already present on machine and can be accessed by the pod diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/gpu-hold.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/gpu-hold.txt new file mode 100644 index 00000000..2c2ee671 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/gpu-hold.txt @@ -0,0 +1,4 @@ +gpu-hold-1 renderD129 0000:48:00.0 +gpu-hold-2 renderD131 0000:96:00.0 +gpu-hold-3 renderD128 0000:12:00.0 +gpu-hold-4 renderD130 0000:7c:00.0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/live.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/live.log new file mode 100644 index 00000000..74383990 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/live.log @@ -0,0 +1,5 @@ +== arm H53 start 00:25:35 (+0s) +== k3d-gpu up 00:25:35 (+0s) +== wait-ready 00:28:27 (+172s) +== cleanup 00:36:21 (+646s) +== done 00:36:23 (+648s) diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.attempt2.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.attempt2.txt new file mode 100644 index 00000000..aaa58210 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.attempt2.txt @@ -0,0 +1,288 @@ +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] █ █ █▄ ▄█ +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.28.0 +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] █▄█▀ █ █ █ █ model /data/Qwen3.8-27B +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:333] +(APIServer pid=1) INFO 10-30 00:32:20 [api_utils.py:272] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_xml', 'host': '0.0.0.0', 'model': '/data/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'float16', 'enforce_eager': True, 'disable_sliding_window': True, 'served_model_name': ['Qwen/Qwen3.6-27B'], 'tensor_parallel_size': 4, 'block_size': 64, 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': False} +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_HOST +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_PORT +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_ATTENTION_BACKEND +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_PORT +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_PORT_VLLM +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_PROTO +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_USE_V1 +(APIServer pid=1) WARNING 10-30 00:32:20 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_ADDR +(APIServer pid=1) INFO 10-30 00:32:20 [model.py:672] Resolved architecture: Qwen3_5ForConditionalGeneration +(APIServer pid=1) WARNING 10-30 00:32:20 [model.py:2299] Casting torch.bfloat16 to torch.float16. +(APIServer pid=1) INFO 10-30 00:32:20 [model.py:1965] Using max model len 262144 +(APIServer pid=1) [ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(APIServer pid=1) [ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(APIServer pid=1) WARNING 10-30 00:32:22 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +(APIServer pid=1) WARNING 10-30 00:32:22 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +(APIServer pid=1) INFO 10-30 00:32:22 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(APIServer pid=1) INFO 10-30 00:32:22 [vllm.py:1533] Cudagraph is disabled under eager mode +(APIServer pid=1) WARNING 10-30 00:32:22 [xpu.py:302] XPU Graph is disabled by environment variable, please set VLLM_XPU_ENABLE_XPU_GRAPH=1 to enable it. +(APIServer pid=1) INFO 10-30 00:32:22 [xpu.py:381] XPU platform: set server shutdown_timeout=5. +(APIServer pid=1) INFO 10-30 00:32:22 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +(APIServer pid=1) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. +(EngineCore pid=242) INFO 10-30 00:32:42 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/data/Qwen3.8-27B', speculative_config=None, tokenizer='/data/Qwen3.8-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=xpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Qwen/Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(EngineCore pid=242) INFO 10-30 00:32:42 [multiproc_executor.py:150] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.42.0.14 (local), world_size=4, local_world_size=4 +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=496) INFO 10-30 00:32:58 [parallel_state.py:1638] world_size=4 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_39d479ea07474d849ac60b9c5c0bfe25 backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=571) INFO 10-30 00:33:11 [parallel_state.py:1638] world_size=4 rank=1 local_rank=1 distributed_init_method=file:///tmp/vllm_dist_39d479ea07474d849ac60b9c5c0bfe25 backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=652) INFO 10-30 00:33:24 [parallel_state.py:1638] world_size=4 rank=2 local_rank=2 distributed_init_method=file:///tmp/vllm_dist_39d479ea07474d849ac60b9c5c0bfe25 backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=733) INFO 10-30 00:33:37 [parallel_state.py:1638] world_size=4 rank=3 local_rank=3 distributed_init_method=file:///tmp/vllm_dist_39d479ea07474d849ac60b9c5c0bfe25 backend=xccl +(Worker pid=496) INFO 10-30 00:33:37 [parallel_state.py:1982] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +2026:10:30-00:33:37: 571 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:33:37: 571 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:33:37: 571 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:33:37: 571 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:33:37: 496 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:33:37: 496 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:33:37: 496 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:33:37: 496 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:33:37: 571 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:33:37: 733 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:33:37: 496 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:33:37: 733 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:33:37: 733 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:33:37: 733 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:33:37: 652 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:33:37: 652 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:33:37: 652 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:33:37: 652 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:33:37: 733 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:33:37: 652 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:33:38: 571:[1] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:33:38: 733:[3] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:33:38: 652:[2] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:33:38: 496:[0] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:33:38: 496:[0] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:33:38: 571:[1] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:33:38: 733:[3] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:33:38: 652:[2] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:33:38: 496:[0] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:33:38: 571:[1] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:33:38: 733:[3] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:33:38: 652:[2] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=733) ERROR 10-30 00:33:38 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:33:38 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:33:38 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=652) ERROR 10-30 00:33:38 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(EngineCore pid=242) INFO 10-30 00:33:38 [multiproc_executor.py:469] [shutdown] Executor: waiting for worker exit count=4 +(EngineCore pid=242) INFO 10-30 00:33:40 [multiproc_executor.py:476] [shutdown] Executor: all workers exited gracefully +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] EngineCore failed to start. +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] Traceback (most recent call last): +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 1315, in run_engine_core +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] return func(*args, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 1073, in __init__ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] super().__init__( +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 133, in __init__ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] self.model_executor = executor_class(vllm_config) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 116, in __init__ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] super().__init__(vllm_config) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] return func(*args, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/abstract.py", line 110, in __init__ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] self._init_executor() +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 211, in _init_executor +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] self.workers = WorkerProc.wait_for_ready(unready_workers) +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 805, in wait_for_ready +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] raise e from None +(EngineCore pid=242) ERROR 10-30 00:33:40 [core.py:1346] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. +(EngineCore pid=242) Process EngineCore: +(EngineCore pid=242) Traceback (most recent call last): +(EngineCore pid=242) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap +(EngineCore pid=242) self.run() +(EngineCore pid=242) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run +(EngineCore pid=242) self._target(*self._args, **self._kwargs) +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1350, in run_engine_core +(EngineCore pid=242) raise e +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1315, in run_engine_core +(EngineCore pid=242) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) return func(*args, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1073, in __init__ +(EngineCore pid=242) super().__init__( +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 133, in __init__ +(EngineCore pid=242) self.model_executor = executor_class(vllm_config) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 116, in __init__ +(EngineCore pid=242) super().__init__(vllm_config) +(EngineCore pid=242) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) return func(*args, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/abstract.py", line 110, in __init__ +(EngineCore pid=242) self._init_executor() +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 211, in _init_executor +(EngineCore pid=242) self.workers = WorkerProc.wait_for_ready(unready_workers) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 805, in wait_for_ready +(EngineCore pid=242) raise e from None +(EngineCore pid=242) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. +(APIServer pid=1) Traceback (most recent call last): +(APIServer pid=1) File "", line 198, in _run_module_as_main +(APIServer pid=1) File "", line 88, in _run_code +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 779, in +(APIServer pid=1) uvloop.run(run_server(args)) +(APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run +(APIServer pid=1) return __asyncio.run( +(APIServer pid=1) ^^^^^^^^^^^^^^ +(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run +(APIServer pid=1) return runner.run(main) +(APIServer pid=1) ^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run +(APIServer pid=1) return self._loop.run_until_complete(task) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete +(APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper +(APIServer pid=1) return await main +(APIServer pid=1) ^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 739, in run_server +(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 753, in run_server_worker +(APIServer pid=1) async with build_async_engine_client( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=1) return await anext(self.gen) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 127, in build_async_engine_client +(APIServer pid=1) async with build_async_engine_client_from_engine_args( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=1) return await anext(self.gen) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 163, in build_async_engine_client_from_engine_args +(APIServer pid=1) async_llm = AsyncLLM.from_vllm_config( +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 220, in from_vllm_config +(APIServer pid=1) return cls( +(APIServer pid=1) ^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 149, in __init__ +(APIServer pid=1) self.engine_core = EngineCoreClient.make_async_mp_client( +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=1) return func(*args, **kwargs) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 139, in make_async_mp_client +(APIServer pid=1) return AsyncMPClient(*client_args) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=1) return func(*args, **kwargs) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 990, in __init__ +(APIServer pid=1) super().__init__( +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 609, in __init__ +(APIServer pid=1) with launch_core_engines( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ +(APIServer pid=1) next(self.gen) +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/utils.py", line 1206, in launch_core_engines +(APIServer pid=1) wait_for_engine_startup( +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/utils.py", line 1286, in wait_for_engine_startup +(APIServer pid=1) raise RuntimeError( +(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.previous.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.previous.txt new file mode 100644 index 00000000..81c267a2 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.previous.txt @@ -0,0 +1,288 @@ +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] █ █ █▄ ▄█ +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.28.0 +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] █▄█▀ █ █ █ █ model /data/Qwen3.8-27B +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:333] +(APIServer pid=1) INFO 10-30 00:34:10 [api_utils.py:272] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_xml', 'host': '0.0.0.0', 'model': '/data/Qwen3.8-27B', 'trust_remote_code': True, 'dtype': 'float16', 'enforce_eager': True, 'disable_sliding_window': True, 'served_model_name': ['Qwen/Qwen3.6-27B'], 'tensor_parallel_size': 4, 'block_size': 64, 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': False} +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_HOST +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_PORT +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_ATTENTION_BACKEND +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_PORT +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_SERVICE_PORT_VLLM +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_PROTO +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_USE_V1 +(APIServer pid=1) WARNING 10-30 00:34:10 [envs.py:2213] Unknown vLLM environment variable detected: VLLM_QWEN38_27B_XPU_SERVICE_PORT_80_TCP_ADDR +(APIServer pid=1) INFO 10-30 00:34:10 [model.py:672] Resolved architecture: Qwen3_5ForConditionalGeneration +(APIServer pid=1) WARNING 10-30 00:34:10 [model.py:2299] Casting torch.bfloat16 to torch.float16. +(APIServer pid=1) INFO 10-30 00:34:10 [model.py:1965] Using max model len 262144 +(APIServer pid=1) [ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(APIServer pid=1) [ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(APIServer pid=1) WARNING 10-30 00:34:12 [vllm.py:1285] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none +(APIServer pid=1) WARNING 10-30 00:34:12 [vllm.py:1354] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. +(APIServer pid=1) INFO 10-30 00:34:12 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) +(APIServer pid=1) INFO 10-30 00:34:12 [vllm.py:1533] Cudagraph is disabled under eager mode +(APIServer pid=1) WARNING 10-30 00:34:12 [xpu.py:302] XPU Graph is disabled by environment variable, please set VLLM_XPU_ENABLE_XPU_GRAPH=1 to enable it. +(APIServer pid=1) INFO 10-30 00:34:12 [xpu.py:381] XPU platform: set server shutdown_timeout=5. +(APIServer pid=1) INFO 10-30 00:34:12 [compilation.py:329] Enabled custom fusions: norm_quant, act_quant +(APIServer pid=1) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`. +(EngineCore pid=242) INFO 10-30 00:34:31 [core.py:122] Initializing a V1 LLM engine (v0.28.0) with config: model='/data/Qwen3.8-27B', speculative_config=None, tokenizer='/data/Qwen3.8-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=xpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Qwen/Qwen3.6-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto') +(EngineCore pid=242) INFO 10-30 00:34:31 [multiproc_executor.py:150] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.42.0.14 (local), world_size=4, local_world_size=4 +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=496) INFO 10-30 00:34:47 [parallel_state.py:1638] world_size=4 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_7e64a13e5b97457ab44674bb3ab7c1fb backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=571) INFO 10-30 00:35:00 [parallel_state.py:1638] world_size=4 rank=1 local_rank=1 distributed_init_method=file:///tmp/vllm_dist_7e64a13e5b97457ab44674bb3ab7c1fb backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=658) INFO 10-30 00:35:13 [parallel_state.py:1638] world_size=4 rank=2 local_rank=2 distributed_init_method=file:///tmp/vllm_dist_7e64a13e5b97457ab44674bb3ab7c1fb backend=xccl +[ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +[ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py. +(Worker pid=739) INFO 10-30 00:35:26 [parallel_state.py:1638] world_size=4 rank=3 local_rank=3 distributed_init_method=file:///tmp/vllm_dist_7e64a13e5b97457ab44674bb3ab7c1fb backend=xccl +(Worker pid=496) INFO 10-30 00:35:26 [parallel_state.py:1982] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A +2026:10:30-00:35:26: 739 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:35:26: 739 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:35:26: 739 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:35:26: 739 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:35:26: 496 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:35:26: 496 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:35:26: 496 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:35:26: 496 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:35:26: 658 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:35:26: 658 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:35:26: 658 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:35:26: 658 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:35:26: 739 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:35:26: 496 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:35:26: 658 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:35:26: 571 |CCL_WARN| value of CCL_WORKER_OFFLOAD changed to be 0 (default:1) +2026:10:30-00:35:26: 571 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi) +2026:10:30-00:35:26: 571 |CCL_WARN| value of CCL_ALLREDUCE changed to be ring (default:) +2026:10:30-00:35:26: 571 |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be none (default:hydra) +2026:10:30-00:35:26: 571 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL +2026:10:30-00:35:27: 496:[0] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:35:27: 658:[2] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:35:27: 571:[1] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:35:27: 739:[3] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices +2026:10:30-00:35:27: 496:[0] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:35:27: 571:[1] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:35:27: 739:[3] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:35:27: 658:[2] |CCL_ERROR| ze_fd_manager.cpp:144 init_device_fds: condition dir failed +opendir failed: could not open device directory +2026:10:30-00:35:27: 496:[0] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:35:27: 571:[1] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:35:27: 739:[3] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +2026:10:30-00:35:27: 658:[2] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=739) ERROR 10-30 00:35:27 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=496) ERROR 10-30 00:35:27 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=571) ERROR 10-30 00:35:27 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] WorkerProc failed to start. +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] Traceback (most recent call last): +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 908, in worker_main +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] worker = WorkerProc(*args, **kwargs) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 669, in __init__ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 335, in init_device +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] self.worker.init_device() # type: ignore +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/workspace/vllm/vllm/v1/worker/xpu_worker.py", line 105, in init_device +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] torch.distributed.all_reduce(torch.zeros(1).xpu()) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] return func(*args, **kwargs) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3245, in all_reduce +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] work = group.allreduce([tensor], opts) +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(Worker pid=658) ERROR 10-30 00:35:27 [multiproc_executor.py:941] RuntimeError: oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory +(EngineCore pid=242) INFO 10-30 00:35:27 [multiproc_executor.py:469] [shutdown] Executor: waiting for worker exit count=4 +(EngineCore pid=242) INFO 10-30 00:35:29 [multiproc_executor.py:476] [shutdown] Executor: all workers exited gracefully +(EngineCore pid=242) Process EngineCore: +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] EngineCore failed to start. +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] Traceback (most recent call last): +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 1315, in run_engine_core +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] return func(*args, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 1073, in __init__ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] super().__init__( +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/engine/core.py", line 133, in __init__ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] self.model_executor = executor_class(vllm_config) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 116, in __init__ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] super().__init__(vllm_config) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] return func(*args, **kwargs) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/abstract.py", line 110, in __init__ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] self._init_executor() +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 211, in _init_executor +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] self.workers = WorkerProc.wait_for_ready(unready_workers) +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 805, in wait_for_ready +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] raise e from None +(EngineCore pid=242) ERROR 10-30 00:35:29 [core.py:1346] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. +(EngineCore pid=242) Traceback (most recent call last): +(EngineCore pid=242) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap +(EngineCore pid=242) self.run() +(EngineCore pid=242) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run +(EngineCore pid=242) self._target(*self._args, **self._kwargs) +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1350, in run_engine_core +(EngineCore pid=242) raise e +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1315, in run_engine_core +(EngineCore pid=242) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) return func(*args, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 1073, in __init__ +(EngineCore pid=242) super().__init__( +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/engine/core.py", line 133, in __init__ +(EngineCore pid=242) self.model_executor = executor_class(vllm_config) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 116, in __init__ +(EngineCore pid=242) super().__init__(vllm_config) +(EngineCore pid=242) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(EngineCore pid=242) return func(*args, **kwargs) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/abstract.py", line 110, in __init__ +(EngineCore pid=242) self._init_executor() +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 211, in _init_executor +(EngineCore pid=242) self.workers = WorkerProc.wait_for_ready(unready_workers) +(EngineCore pid=242) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(EngineCore pid=242) File "/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 805, in wait_for_ready +(EngineCore pid=242) raise e from None +(EngineCore pid=242) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. +(APIServer pid=1) Traceback (most recent call last): +(APIServer pid=1) File "", line 198, in _run_module_as_main +(APIServer pid=1) File "", line 88, in _run_code +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 779, in +(APIServer pid=1) uvloop.run(run_server(args)) +(APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run +(APIServer pid=1) return __asyncio.run( +(APIServer pid=1) ^^^^^^^^^^^^^^ +(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run +(APIServer pid=1) return runner.run(main) +(APIServer pid=1) ^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run +(APIServer pid=1) return self._loop.run_until_complete(task) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete +(APIServer pid=1) File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper +(APIServer pid=1) return await main +(APIServer pid=1) ^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 739, in run_server +(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 753, in run_server_worker +(APIServer pid=1) async with build_async_engine_client( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=1) return await anext(self.gen) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 127, in build_async_engine_client +(APIServer pid=1) async with build_async_engine_client_from_engine_args( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=1) return await anext(self.gen) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 163, in build_async_engine_client_from_engine_args +(APIServer pid=1) async_llm = AsyncLLM.from_vllm_config( +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 220, in from_vllm_config +(APIServer pid=1) return cls( +(APIServer pid=1) ^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 149, in __init__ +(APIServer pid=1) self.engine_core = EngineCoreClient.make_async_mp_client( +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=1) return func(*args, **kwargs) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 139, in make_async_mp_client +(APIServer pid=1) return AsyncMPClient(*client_args) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=1) return func(*args, **kwargs) +(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 990, in __init__ +(APIServer pid=1) super().__init__( +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/core_client.py", line 609, in __init__ +(APIServer pid=1) with launch_core_engines( +(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ +(APIServer pid=1) next(self.gen) +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/utils.py", line 1206, in launch_core_engines +(APIServer pid=1) wait_for_engine_startup( +(APIServer pid=1) File "/workspace/vllm/vllm/v1/engine/utils.py", line 1286, in wait_for_engine_startup +(APIServer pid=1) raise RuntimeError( +(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/log-vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz.txt new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pin.yaml b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pin.yaml new file mode 100644 index 00000000..96094b70 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pin.yaml @@ -0,0 +1,2 @@ +image: + tag: "v0.28.0@sha256:4756b66a077627133cee653b551f6f5eaa1b9a981b5eea13edd33fcd3b0d3ca3" diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pods.txt b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pods.txt new file mode 100644 index 00000000..571eb094 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/pods.txt @@ -0,0 +1,10 @@ +NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +default vllm-qwen38-27b-xpu-69c59cbbdc-5zbzz 0/1 Running 3 (32s ago) 7m37s 10.42.0.14 k3d-k3dgpu-e-h53-server-0 +kube-system coredns-577d995dff-bsdh4 1/1 Running 0 10m 10.42.0.2 k3d-k3dgpu-e-h53-server-0 +kube-system helm-install-gateway-api-crd-n4z4l 0/1 Completed 0 10m 10.42.0.9 k3d-k3dgpu-e-h53-server-0 +kube-system intel-gpu-plugin-sfv7x 1/1 Running 0 10m 10.42.0.8 k3d-k3dgpu-e-h53-server-0 +kube-system local-path-provisioner-6858d854cf-wx2r9 1/1 Running 0 10m 10.42.0.3 k3d-k3dgpu-e-h53-server-0 +kube-system metrics-server-778c4649b4-gp9l4 1/1 Running 0 10m 10.42.0.5 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-gc-7d6955597d-28trj 1/1 Running 0 10m 10.42.0.7 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-master-74b6cfb7df-6wq4q 1/1 Running 0 10m 10.42.0.4 k3d-k3dgpu-e-h53-server-0 +node-feature-discovery nfd-worker-j8s9j 1/1 Running 0 10m 10.42.0.6 k3d-k3dgpu-e-h53-server-0 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/reset-mcp.sh b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/reset-mcp.sh new file mode 100755 index 00000000..b21a5876 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/reset-mcp.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +kill "$(cat "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold/mcp.pid")" 2>/dev/null +for _ in $(seq 50); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null || break; sleep 0.1; done +cd "/home/devcloud/k3dgpu-bench/eab/usecases/banking-assistant/src" && (MCP_HOST=127.0.0.1 MCP_PORT=18600 exec "/home/devcloud/k3dgpu-bench/mafvenv/bin/python" -m mcp_server.server >> "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold/mcp.log" 2>&1 & + echo $! > "/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold/mcp.pid") +for _ in $(seq 120); do (exec 3<>"/dev/tcp/127.0.0.1/18600") 2>/dev/null && exit 0; sleep 0.5; done +exit 1 diff --git a/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/up.log b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/up.log new file mode 100644 index 00000000..01829362 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/smoke/H53-smoke-B2-b/hold/up.log @@ -0,0 +1,33 @@ +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators: intel amx +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-e-h53' with image cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-e-h53' +INFO[0000] Created image volume k3d-k3dgpu-e-h53-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-e-h53-tools' +INFO[0001] Creating node 'k3d-k3dgpu-e-h53-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-e-h53' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-e-h53-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-e-h53' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-e-h53) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold/kubeconfig +==> Waiting for nodes Ready... +node/k3d-k3dgpu-e-h53-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +==> intel device plugin auto-deployed by k3s. +==> Verifying accelerators are schedulable... +Waiting for daemon set "intel-gpu-plugin" rollout to finish: 0 of 1 updated pods are available... +daemon set "intel-gpu-plugin" successfully rolled out +WARN: present but disabled: DSA dsa0 dsa2 dsa4, QAT 0000:01:00.0 0000:06:00.0 0000:0b:00.0 (no user work queues / VFs), so not in the cluster — enable them: k3d-gpu host-setup --apply, then k3d-gpu down && k3d-gpu up (exact commands: k3d-gpu doctor) +==> Accelerators schedulable: intel=4 amx=1. Try: k3d-gpu test diff --git a/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-AG.launch.log b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-AG.launch.log new file mode 100644 index 00000000..9b222c03 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-AG.launch.log @@ -0,0 +1,11 @@ +2026-10-30T00:15:13Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:15:13Z smoke H53-smoke-AG start +== H53-smoke-AG waiting for the lock 2026-10-30T00:15:13Z +== H53-smoke-AG lock taken 2026-10-30T00:15:13Z +== H53-smoke-AG start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=AG LIVE_CAP=2370 LIVE_HOLD=1 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER= LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold 2026-10-30T00:15:14Z +/home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/crescent/hold.sh: line 151: 1549367 Killed ( while kill -0 "${cp}" 2> /dev/null; do + sleep 30; if [ "$(disk)" -gt 97 ]; then + echo "disk $(disk)% at $(date -u +%FT%TZ)" > "${R}/disk-abort"; kill -TERM "$$"; break; + fi; +done ) 7>&- 8>&- 9>&- +== H53-smoke-AG rc=0; clusters deleted: none; containers removed: none; now: 2026-10-30T00:20:05Z diff --git a/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-a.launch.log b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-a.launch.log new file mode 100644 index 00000000..64115dbd --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-a.launch.log @@ -0,0 +1,11 @@ +2026-10-30T00:20:14Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:20:14Z smoke H53-smoke-B2-a start +== H53-smoke-B2-a waiting for the lock 2026-10-30T00:20:14Z +== H53-smoke-B2-a lock taken 2026-10-30T00:20:14Z +== H53-smoke-B2-a start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=B2 LIVE_CAP=2370 LIVE_HOLD=2 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER= LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold 2026-10-30T00:20:14Z +/home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/crescent/hold.sh: line 151: 1565528 Killed ( while kill -0 "${cp}" 2> /dev/null; do + sleep 30; if [ "$(disk)" -gt 97 ]; then + echo "disk $(disk)% at $(date -u +%FT%TZ)" > "${R}/disk-abort"; kill -TERM "$$"; break; + fi; +done ) 7>&- 8>&- 9>&- +== H53-smoke-B2-a rc=143; clusters deleted: none; containers removed: none; now: 2026-10-30T00:25:26Z diff --git a/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-b.launch.log b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-b.launch.log new file mode 100644 index 00000000..2ff4c305 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/H53-smoke-B2-b.launch.log @@ -0,0 +1,11 @@ +2026-10-30T00:25:35Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:25:35Z smoke H53-smoke-B2-b start +== H53-smoke-B2-b waiting for the lock 2026-10-30T00:25:35Z +== H53-smoke-B2-b lock taken 2026-10-30T00:25:35Z +== H53-smoke-B2-b start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=B2 LIVE_CAP=2370 LIVE_HOLD=2 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER=b LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold 2026-10-30T00:25:35Z +/home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/crescent/hold.sh: line 151: 1579257 Killed ( while kill -0 "${cp}" 2> /dev/null; do + sleep 30; if [ "$(disk)" -gt 97 ]; then + echo "disk $(disk)% at $(date -u +%FT%TZ)" > "${R}/disk-abort"; kill -TERM "$$"; break; + fi; +done ) 7>&- 8>&- 9>&- +== H53-smoke-B2-b rc=143; clusters deleted: none; containers removed: none; now: 2026-10-30T00:36:28Z diff --git a/bench/eatflow/results/2026-10-20/H53/window/campaign-H53.log b/bench/eatflow/results/2026-10-20/H53/window/campaign-H53.log new file mode 100644 index 00000000..4fa8d93a --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/campaign-H53.log @@ -0,0 +1,12 @@ +== H53-smoke-AG waiting for the lock 2026-10-30T00:15:13Z +== H53-smoke-AG lock taken 2026-10-30T00:15:13Z +== H53-smoke-AG start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=AG LIVE_CAP=2370 LIVE_HOLD=1 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER= LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-AG/hold 2026-10-30T00:15:14Z +== H53-smoke-AG rc=0; clusters deleted: none; containers removed: none; now: 2026-10-30T00:20:05Z +== H53-smoke-B2-a waiting for the lock 2026-10-30T00:20:14Z +== H53-smoke-B2-a lock taken 2026-10-30T00:20:14Z +== H53-smoke-B2-a start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=B2 LIVE_CAP=2370 LIVE_HOLD=2 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER= LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-a/hold 2026-10-30T00:20:14Z +== H53-smoke-B2-a rc=143; clusters deleted: none; containers removed: none; now: 2026-10-30T00:25:26Z +== H53-smoke-B2-b waiting for the lock 2026-10-30T00:25:35Z +== H53-smoke-B2-b lock taken 2026-10-30T00:25:35Z +== H53-smoke-B2-b start: env K3D_GPU_IMAGE=cryptoandcoffee/k3d-gpu:0.1.61@sha256:8cd264aa68c77681d7fa474693944de0db8d55de5881253e4029bf72a67e175c K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1% LIVE_MODE=solo LIVE_SOLO=B2 LIVE_CAP=2370 LIVE_HOLD=2 LIVE_PAIRS_GUARD=99999 LIVE_PAIRS_DIR=/home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/pairs LIVE_MODEL_SET=qwen38 LIVE_MOUNT_B=1 LIVE_GPU_FREE=0000:12:00.0 0000:48:00.0 0000:7c:00.0 0000:96:00.0 LIVE_B2_TP=4 LIVE_B2_LADDER=b LIVE_EAT_XPU_CHART=/home/devcloud/k3dgpu-bench/eat-pr21/core/helm-charts/vllm LIVE_MODELS=/home/devcloud/k3dgpu-bench/models LIVE_BLUEPRINT=/home/devcloud/k3dgpu-bench/eab LIVE_PY=/home/devcloud/k3dgpu-bench/mafvenv/bin/python PATH=/home/devcloud/k3dgpu-bench/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin bash /home/devcloud/k3dgpu-bench/k3d-gpu-h53/bench/eatflow/live.sh H53 /home/devcloud/k3dgpu-bench/smoke/H53/H53-smoke-B2-b/hold 2026-10-30T00:25:35Z +== H53-smoke-B2-b rc=143; clusters deleted: none; containers removed: none; now: 2026-10-30T00:36:28Z diff --git a/bench/eatflow/results/2026-10-20/H53/window/deadman.log b/bench/eatflow/results/2026-10-20/H53/window/deadman.log new file mode 100644 index 00000000..e69de29b diff --git a/bench/eatflow/results/2026-10-20/H53/window/events.log b/bench/eatflow/results/2026-10-20/H53/window/events.log new file mode 100644 index 00000000..166a365c --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/events.log @@ -0,0 +1,23 @@ +2026-10-30T00:14:50Z VRAM before the stop: 0000:12:00.0 16.14;0000:48:00.0 15.43;0000:7c:00.0 16.18;0000:96:00.0 15.47; +2026-10-30T00:15:00Z qwen38-a-hr stopped +2026-10-30T00:15:06Z qwen38-b-hr stopped +2026-10-30T00:15:06Z window open; deadman pid 1548781 +2026-10-30T00:15:06Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.17;0000:96:00.0 0.0; +2026-10-30T00:15:06Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.17;0000:96:00.0 0.0; +2026-10-30T00:15:13Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:15:13Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:15:13Z smoke H53-smoke-AG start +2026-10-30T00:20:14Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:20:14Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:20:14Z smoke H53-smoke-B2-a start +2026-10-30T00:25:19Z smoke H53-smoke-B2-a early stop: setvars.sh missing, 2 restarts (R1 item 4); TERM to hold.sh pid 1565186 +2026-10-30T00:25:35Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:25:35Z cards free: 0000:12:00.0 0.0;0000:48:00.0 0.0;0000:7c:00.0 0.0;0000:96:00.0 0.0; +2026-10-30T00:25:35Z smoke H53-smoke-B2-b start +2026-10-30T00:36:21Z smoke H53-smoke-B2-b early stop: oneCCL init_device_fds opendir failed (/dev/dri/by-path absent in the k3d node and pod), the same error at 00:33:38Z and 00:35:27Z (R1 item 4); TERM to hold.sh pid 1578915 +2026-10-30T00:36:38Z B2 ladder: rung (b) fails with an error no further rung addresses ((c) --disable-log-requests is absent; (d) is for a KV-cache failure): B2 not deliverable, no hold runs (H53 pre-registration item 3, R1 item 4); restoring serving +2026-10-30T00:37:45Z qwen38-a-hr running=true; qwen38-a-hr: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:45Z qwen38-a-hr: image, command, env, devices, mounts and ports as before the stop +2026-10-30T00:37:47Z qwen38-b-hr running=true; qwen38-b-hr: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:47Z qwen38-b-hr: image, command, env, devices, mounts and ports as before the stop +2026-10-30T00:38:03Z window closed (restore rc=0) diff --git a/bench/eatflow/results/2026-10-20/H53/window/holds-H53.log b/bench/eatflow/results/2026-10-20/H53/window/holds-H53.log new file mode 100644 index 00000000..e58209b1 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/holds-H53.log @@ -0,0 +1,6 @@ +{"ev": "start", "host": "905a08313800", "name": "H53-smoke-AG", "t": 1793319313.776, "tag": "H53-smoke-AG:1549024:380698146"} +{"ev": "end", "host": "905a08313800", "orphans_killed": "0", "rc": "0", "reap_residue": "0", "stale_reaped": "0", "t": 1793319605.845, "tag": "H53-smoke-AG:1549024:380698146"} +{"ev": "start", "host": "905a08313800", "name": "H53-smoke-B2-a", "t": 1793319614.615, "tag": "H53-smoke-B2-a:1565186:380728230"} +{"ev": "end", "host": "905a08313800", "orphans_killed": "0", "rc": "143", "reap_residue": "0", "stale_reaped": "0", "t": 1793319926.825, "tag": "H53-smoke-B2-a:1565186:380728230"} +{"ev": "start", "host": "905a08313800", "name": "H53-smoke-B2-b", "t": 1793319935.15, "tag": "H53-smoke-B2-b:1578915:380760283"} +{"ev": "end", "host": "905a08313800", "orphans_killed": "0", "rc": "143", "reap_residue": "0", "stale_reaped": "0", "t": 1793320588.769, "tag": "H53-smoke-B2-b:1578915:380760283"} diff --git a/bench/eatflow/results/2026-10-20/H53/window/restore.log b/bench/eatflow/results/2026-10-20/H53/window/restore.log new file mode 100644 index 00000000..1950dd11 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/restore.log @@ -0,0 +1,4 @@ +2026-10-30T00:36:38Z qwen38-a-hr started +2026-10-30T00:36:38Z qwen38-b-hr started +2026-10-30T00:37:40Z qwen38-b-hr answering: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:43Z qwen38-a-hr answering: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " diff --git a/bench/eatflow/results/2026-10-20/H53/window/serving-events.log b/bench/eatflow/results/2026-10-20/H53/window/serving-events.log new file mode 100644 index 00000000..5abd3286 --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/serving-events.log @@ -0,0 +1,8 @@ +2026-10-30T00:14:52Z qwen38-a-hr before stop: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:15:00Z qwen38-a-hr stopped (ZE_AFFINITY_MASK=0,1, port 30000) +2026-10-30T00:15:01Z qwen38-b-hr before stop: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:15:06Z qwen38-b-hr stopped (ZE_AFFINITY_MASK=2,3, port 30001) +2026-10-30T00:36:38Z qwen38-a-hr started +2026-10-30T00:36:38Z qwen38-b-hr started +2026-10-30T00:37:40Z qwen38-b-hr answering: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:43Z qwen38-a-hr answering: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " diff --git a/bench/eatflow/results/2026-10-20/H53/window/start.launch.log b/bench/eatflow/results/2026-10-20/H53/window/start.launch.log new file mode 100644 index 00000000..12a5704d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/start.launch.log @@ -0,0 +1,7 @@ +2026-10-30T00:37:45Z qwen38-a-hr running=true; qwen38-a-hr: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:45Z qwen38-a-hr: image, command, env, devices, mounts and ports as before the stop +2026-10-30T00:37:47Z qwen38-b-hr running=true; qwen38-b-hr: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:37:47Z qwen38-b-hr: image, command, env, devices, mounts and ports as before the stop +2026-10-30T00:37:52Z :30000 pass=1 before=2 after=3 content=OK +2026-10-30T00:38:03Z :30001 pass=1 before=3 after=4 content=OK +2026-10-30T00:38:03Z window closed (restore rc=0) diff --git a/bench/eatflow/results/2026-10-20/H53/window/stop.log b/bench/eatflow/results/2026-10-20/H53/window/stop.log new file mode 100644 index 00000000..c509cd0d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/stop.log @@ -0,0 +1,4 @@ +2026-10-30T00:14:52Z qwen38-a-hr before stop: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:15:00Z qwen38-a-hr stopped (ZE_AFFINITY_MASK=0,1, port 30000) +2026-10-30T00:15:01Z qwen38-b-hr before stop: health 200, model Qwen3.8-27B, reply: We need to respond to user: "Say OK". Need final likely just " +2026-10-30T00:15:06Z qwen38-b-hr stopped (ZE_AFFINITY_MASK=2,3, port 30001) diff --git a/bench/eatflow/results/2026-10-20/H53/window/xe-prom.json b/bench/eatflow/results/2026-10-20/H53/window/xe-prom.json new file mode 100644 index 00000000..1017281d --- /dev/null +++ b/bench/eatflow/results/2026-10-20/H53/window/xe-prom.json @@ -0,0 +1 @@ +{"t0": 1793319300, "t1": 1793320690, "note": "read-only query_range of crescent Prometheus after the window, step 15 s", "xe_vram_used_bytes": [{"metric": {"__name__": "xe_vram_used_bytes", "bdf": "0000:12:00.0", "card": "card1", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "181055488"], [1793320275, "181055488"], [1793320290, "362127360"], [1793320305, "543207424"], [1793320320, "905367552"], [1793320335, "0"], [1793320350, "181051392"], [1793320365, "181051392"], [1793320380, "362131456"], [1793320395, "543223808"], [1793320410, "724287488"], [1793320425, "0"], [1793320440, "0"], [1793320455, "181055488"], [1793320470, "181055488"], [1793320485, "181055488"], [1793320500, "543211520"], [1793320515, "724271104"], [1793320530, "181055488"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "181055488"], [1793320635, "543129600"], [1793320650, "27628863488"], [1793320665, "28720324608"], [1793320680, "28722880512"]]}, {"metric": {"__name__": "xe_vram_used_bytes", "bdf": "0000:48:00.0", "card": "card2", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "2150400"], [1793320275, "2150400"], [1793320290, "183209984"], [1793320305, "364261376"], [1793320320, "547430400"], [1793320335, "0"], [1793320350, "2134016"], [1793320365, "2134016"], [1793320380, "183205888"], [1793320395, "364249088"], [1793320410, "545284096"], [1793320425, "0"], [1793320440, "0"], [1793320455, "2125824"], [1793320470, "2125824"], [1793320485, "2125824"], [1793320500, "185303040"], [1793320515, "545308672"], [1793320530, "2125824"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "2101248"], [1793320635, "185241600"], [1793320650, "15433220096"], [1793320665, "16510111744"], [1793320680, "16512667648"]]}, {"metric": {"__name__": "xe_vram_used_bytes", "bdf": "0000:7c:00.0", "card": "card3", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "17368285184"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "6586368"], [1793319540, "1816489984"], [1793319555, "15901691904"], [1793319570, "15901642752"], [1793319585, "15901642752"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "2105344"], [1793320275, "2105344"], [1793320290, "183152640"], [1793320305, "364199936"], [1793320320, "547348480"], [1793320335, "0"], [1793320350, "2117632"], [1793320365, "2117632"], [1793320380, "183156736"], [1793320395, "364199936"], [1793320410, "545243136"], [1793320425, "0"], [1793320440, "0"], [1793320455, "2113536"], [1793320470, "2113536"], [1793320485, "2113536"], [1793320500, "185266176"], [1793320515, "545251328"], [1793320530, "2113536"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "181043200"], [1793320635, "543141888"], [1793320650, "27856384000"], [1793320665, "28471382016"], [1793320680, "28471840768"]]}, {"metric": {"__name__": "xe_vram_used_bytes", "bdf": "0000:96:00.0", "card": "card4", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "16610652160"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "2113536"], [1793320275, "2113536"], [1793320290, "183156736"], [1793320305, "364199936"], [1793320320, "547348480"], [1793320335, "0"], [1793320350, "2105344"], [1793320365, "2105344"], [1793320380, "183144448"], [1793320395, "364183552"], [1793320410, "545226752"], [1793320425, "0"], [1793320440, "0"], [1793320455, "2109440"], [1793320470, "2109440"], [1793320485, "2109440"], [1793320500, "185253888"], [1793320515, "545230848"], [1793320530, "2109440"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "2105344"], [1793320635, "185241600"], [1793320650, "15646273536"], [1793320665, "16261324800"], [1793320680, "16261787648"]]}], "xe_engine_utilization_percent": [{"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "bcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "1.11"], [1793320665, "12.37"], [1793320680, "3.76"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "ccs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "15"], [1793320665, "24.15"], [1793320680, "9.59"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "rcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "vcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:12:00.0", "card": "card1", "engine": "vecs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "bcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "1.08"], [1793320665, "11.3"], [1793320680, "3.6"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "ccs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "15.96"], [1793320665, "22.18"], [1793320680, "9.58"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "rcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "vcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:48:00.0", "card": "card2", "engine": "vecs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "bcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "1.97"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "45.46"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "1.72"], [1793320665, "9.84"], [1793320680, "2.01"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "ccs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "5.8"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "19.42"], [1793320665, "14.44"], [1793320680, "6"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "rcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "vcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:7c:00.0", "card": "card3", "engine": "vecs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "bcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "2.01"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "1.83"], [1793320665, "9.88"], [1793320680, "2.27"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "ccs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "5.8"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "19.05"], [1793320665, "14.53"], [1793320680, "6.02"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "rcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "vcs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}, {"metric": {"__name__": "xe_engine_utilization_percent", "bdf": "0000:96:00.0", "card": "card4", "engine": "vecs", "instance": "xe-exporter:9101", "job": "xe-gpu"}, "values": [[1793319300, "0"], [1793319315, "0"], [1793319330, "0"], [1793319345, "0"], [1793319360, "0"], [1793319375, "0"], [1793319390, "0"], [1793319405, "0"], [1793319420, "0"], [1793319435, "0"], [1793319450, "0"], [1793319465, "0"], [1793319480, "0"], [1793319495, "0"], [1793319510, "0"], [1793319525, "0"], [1793319540, "0"], [1793319555, "0"], [1793319570, "0"], [1793319585, "0"], [1793319600, "0"], [1793319615, "0"], [1793319630, "0"], [1793319645, "0"], [1793319660, "0"], [1793319675, "0"], [1793319690, "0"], [1793319705, "0"], [1793319720, "0"], [1793319735, "0"], [1793319750, "0"], [1793319765, "0"], [1793319780, "0"], [1793319795, "0"], [1793319810, "0"], [1793319825, "0"], [1793319840, "0"], [1793319855, "0"], [1793319870, "0"], [1793319885, "0"], [1793319900, "0"], [1793319915, "0"], [1793319930, "0"], [1793319945, "0"], [1793319960, "0"], [1793319975, "0"], [1793319990, "0"], [1793320005, "0"], [1793320020, "0"], [1793320035, "0"], [1793320050, "0"], [1793320065, "0"], [1793320080, "0"], [1793320095, "0"], [1793320110, "0"], [1793320125, "0"], [1793320140, "0"], [1793320155, "0"], [1793320170, "0"], [1793320185, "0"], [1793320200, "0"], [1793320215, "0"], [1793320230, "0"], [1793320245, "0"], [1793320260, "0"], [1793320275, "0"], [1793320290, "0"], [1793320305, "0"], [1793320320, "0"], [1793320335, "0"], [1793320350, "0"], [1793320365, "0"], [1793320380, "0"], [1793320395, "0"], [1793320410, "0"], [1793320425, "0"], [1793320440, "0"], [1793320455, "0"], [1793320470, "0"], [1793320485, "0"], [1793320500, "0"], [1793320515, "0"], [1793320530, "0"], [1793320545, "0"], [1793320560, "0"], [1793320575, "0"], [1793320590, "0"], [1793320605, "0"], [1793320620, "0"], [1793320635, "0"], [1793320650, "0"], [1793320665, "0"], [1793320680, "0"]]}]} \ No newline at end of file diff --git a/docs/intel/technical-report.md b/docs/intel/technical-report.md index 48b1d3e1..e0c17a45 100644 --- a/docs/intel/technical-report.md +++ b/docs/intel/technical-report.md @@ -6,12 +6,12 @@ Status: working report, 2026-10-29. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.m k3d-gpu turns one Linux host into a single-node k3s cluster (k3d) that detects the host's Intel hardware and configures it. It sets NRI balloons for hybrid P/E and NUMA CPUs, device plugins for iGPU, dGPU, NPU, DSA, IAA, QAT and Gaudi, and placement labels for agent and inference pods. -This report covers what was measured, and how. Every result but H52, H52b, H25x, H27x, H28x, H34x, H50, H50o, H51, H54a, H54d, H54b, H54c and H56 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. +This report covers what was measured, and how. Every result but H52, H52b, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54b, H54c and H56 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. - Seventeen hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x is H25 replicated on the Xeon 6741P. H57dm (the Gemma 4 QAT MTP drafter speeds llama.cpp decode on the P-cores) is the newest. - Seventeen are REFUTED. Six of them changed shipped defaults. The newest, H52b, is H52 after `k3d-gpu host-setup --apply` on the Xeon 6741P: DSA was advertised and reached a pod, but IAA got no work queue, the QAT plugin advertised none of the VFs, and `up` advertised two of the four B70s: the serving pairs hold about half the VRAM of every card, and its busy-card filter (half or more) left out the two just over half and kept the two just under it [ledger:H52b]. Before it, H51 ran the graded agents-per-box ramp on the Xeon 6741P: the k3d-gpu CPU stack failed at one agent (turn p95 10.41 s against the 10 s SLO), and so did EAT's stock vLLM CPU arm [ledger:H51]. H50 ran inside EAT's banking flow on the same host: EAT's current stock vLLM CPU arm (release-1.1.0) delivered at least as many tasks within the turn budget as the k3d-gpu CPU stack: 21 of 24 against 19 of 24 [ledger:H50]. -- 28 are INCONCLUSIVE, mostly because the shared host throttles or its load was unmatched. H50o (the k3d-gpu CPU stack against EAT's own OVMS chart on the Xeon 6741P) is INCONCLUSIVE because that chart's OVMS 2025.4 crashed on loading Gemma 4 26B-A4B, so no pair ran. H54a and H54d (Gemma 4 26B-A4B QAT on the Xeon 6741P) are INCONCLUSIVE because the stock arm, EAT 1.0.3's vLLM CPU v0.24.0, as this harness rendered EAT's entry (no CPU balloon, requests or limits), stalled in both of H54a's holds, so no arm could be scored against it. EAT 1.1.0's vLLM CPU v0.28.0 was not measured in H54a; H50 ran it in EAT's banking flow. H54b and H54c (one Arc Pro B70 against two, and experts in DRAM with the rest on one B70) are INCONCLUSIVE because their "one card" pods saw all four cards, so no arm was delivered as pre-registered. H57da (k3d-gpu's OVMS pick vs llama.cpp for Gemma 4 QAT on the i9-13900T) did not resolve: the strictly pinned llama.cpp arms tracked host load; H57db did not run (memory). H28x (H28 replicated on the Xeon 6741P) is INCONCLUSIVE: its one pre-registered hold failed in setup and measured nothing; one re-run is registered. H56 (two holds at once in cpuset slices of the Xeon 6741P) is INCONCLUSIVE: in every hold with a sibling, the sibling's banking block ended shortly after the measured block began, so all four were VOID and no interference statistic was scored; crescent rows stay at one hold at a time. -- 19 are PENDING: the i9-13900T thread-pinning follow-ups H57dp and H57de, the Xeon 6741P rows H53, H54q, H60 (Qwen3.8-Flash-Next from host DRAM) and the zero-touch thesis row H58, the cross-host replication H7x, the engine-choice rows H59 (Xeon 6741P), H59d (i9-13900T) and H59w (i7-13700), H55 (SLO-gated admission of an E-core neighbour on the i9-13900T), and the second client host's (wildct, i7-13700) rows H52w, H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. +- 29 are INCONCLUSIVE, mostly because the shared host throttles or its load was unmatched. H50o (the k3d-gpu CPU stack against EAT's own OVMS chart on the Xeon 6741P) is INCONCLUSIVE because that chart's OVMS 2025.4 crashed on loading Gemma 4 26B-A4B, so no pair ran. H54a and H54d (Gemma 4 26B-A4B QAT on the Xeon 6741P) are INCONCLUSIVE because the stock arm, EAT 1.0.3's vLLM CPU v0.24.0, as this harness rendered EAT's entry (no CPU balloon, requests or limits), stalled in both of H54a's holds, so no arm could be scored against it. EAT 1.1.0's vLLM CPU v0.28.0 was not measured in H54a; H50 ran it in EAT's banking flow. H54b and H54c (one Arc Pro B70 against two, and experts in DRAM with the rest on one B70) are INCONCLUSIVE because their "one card" pods saw all four cards, so no arm was delivered as pre-registered. H53 (EAT's banking flow on the B70s: k3d-gpu's one-card OVMS pick against EAT 1.1.0's vLLM XPU path across all four cards) is INCONCLUSIVE because the EAT arm did not start in a k3d-gpu cluster: oneCCL needs `/dev/dri/by-path` for its cross-card collective, and the k3d node has none, so no hold ran. H57da (k3d-gpu's OVMS pick vs llama.cpp for Gemma 4 QAT on the i9-13900T) did not resolve: the strictly pinned llama.cpp arms tracked host load; H57db did not run (memory). H28x (H28 replicated on the Xeon 6741P) is INCONCLUSIVE: its one pre-registered hold failed in setup and measured nothing; one re-run is registered. H56 (two holds at once in cpuset slices of the Xeon 6741P) is INCONCLUSIVE: in every hold with a sibling, the sibling's banking block ended shortly after the measured block began, so all four were VOID and no interference statistic was scored; crescent rows stay at one hold at a time. +- 18 are PENDING: the i9-13900T thread-pinning follow-ups H57dp and H57de, the Xeon 6741P rows H54q, H60 (Qwen3.8-Flash-Next from host DRAM) and the zero-touch thesis row H58, the cross-host replication H7x, the engine-choice rows H59 (Xeon 6741P), H59d (i9-13900T) and H59w (i7-13700), H55 (SLO-gated admission of an E-core neighbour on the i9-13900T), and the second client host's (wildct, i7-13700) rows H52w, H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. - Every other Intel family is detected and configured in fixture tests only. None has a measurement. No world-first, superlative or priority claim is made; [docs/provenance.md](../provenance.md) supports none. @@ -108,6 +108,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p | H54d | Not run: H54a chose no arm to put in EAT's banking flow [ledger:H54d] | Its arm A is undefined and its arm B is the arm that stalled | | H56 | Two A1 banking holds at once in disjoint cpuset slices of the Xeon 6741P. All 8 holds ran; the 4 holds without a sibling are valid. No interference finding [ledger:H56] | Every hold with a sibling is VOID: a B24 block ends at its 24th task, so the sibling's block ended shortly after the measured one began. 4 VOID exceed the cap of 2 | | H54b | Qwen3.8-27B on Arc Pro B70: the best one-card arm against EAT #21's two-card XPU path. Nothing one-card was delivered: the arms on Qwen bf16 or RedHat int4 weights did not fit the disk, two OVMS arms failed on runner bugs, and llama.cpp SYCL ran on all four cards (direction only) [ledger:H54b] | Placement failure (the pinned pod saw every card) and no gate reference | +| H53 | EAT's banking flow on four Arc Pro B70s with Qwen3.8-27B: k3d-gpu's own GPU pick (OVMS int4 on one card) deployed and answered its warm-up. EAT 1.1.0's vLLM XPU path across four cards needed one fix to start (its command sources a oneAPI script the image lacks), then failed in its first cross-card collective: oneCCL opens `/dev/dri/by-path`, which the k3d node and its pods do not have [ledger:H53] | Not deliverable: the EAT arm's pre-registered boot ladder ran out, so no hold ran. The `/dev/dri/by-path` gap is k3d-gpu's | | H54c | Gemma 4 QAT with routed experts in the Xeon's DRAM and the rest on a B70, against a byte-split model. The experts did run on the Xeon, but every GPU arm split its layers over four cards, so the one-card model was never tested (direction only) [ledger:H54c] | Placement failure; the CPU-only input is confounded too | The external bake-offs are in the tables above: the Ollama, llama.cpp and plain OVMS ones are all decided.