Skip to content

ledger(H53): INCONCLUSIVE, B2 not deliverable on k3d-gpu; amendment R1 and the run's records - #267

Merged
88plug merged 3 commits into
mainfrom
claude/h53-run
Oct 30, 2026
Merged

88plug merged 3 commits into
mainfrom
claude/h53-run

Conversation

@88plug

@88plug 88plug commented Oct 30, 2026

Copy link
Copy Markdown
Owner

H53 ran on crescent on 2026-10-30, from 1f39488 (this branch's first commit, pushed at 00:13:25Z, before the first H53 step). Its verdict is INCONCLUSIVE (not deliverable). No scored hold ran, so 0 of the 8 permitted holds were used.

Amendment R1 (mechanics only; no arm, prediction, statistic, gate, threshold, seed or order changes, and no cap rises):

  • live.sh: stage_path moves to top level. The solo path called it while it was defined only inside the pairs branch, so AG would have served an empty hostPath. A regression test fails on 74ff63e.
  • live.sh: LIVE_PAIRS_GUARD passes through to agentbench solo --guard-s, so a wiring smoke runs only the warm-up. The scored holds leave it unset.
  • solo.py: alive_blocks gives a futility stop. Once fewer than three blocks can still count, the verdict is fixed at INCONCLUSIVE, so no further hold runs. It reads block counts only, never d. Tests are in AliveBlocksTest.
  • prehold.sh covers the window steps: E4's setup.sh under the bench lock, a model re-verify, front-door checks, stop and start through gpu-serving.sh (docker stop, never rm), the AG and B2 smokes, the scored holds, per-card Prometheus records, and a deadman that restores serving if the session dies.
  • Tools follow amendment E4: k3d v5.9.0, kubectl v1.37.2, helm v4.3.1 and uv 0.13.6, all checksummed (prehold/setup.txt).

What happened in the window (00:15 to 00:38Z):

  • AG wiring smoke: passed. agent manifest chose the GPU, and OVMS 2026.4.1 was ready on one card (0000:7c:00.0) and answered its warm-up in 5.5 s. The other three cards stayed at 0 GiB.
  • B2 rung (a): stops at once because /opt/intel/oneapi/setvars.sh is not in vllm-openai-xpu:v0.28.0. The pre-registration had already read this from the image.
  • B2 rung (b): vLLM v0.28.0 starts. Its four XPU workers then fail in their first all_reduce with oneCCL: ze_fd_manager.cpp:144 init_device_fds: opendir failed: could not open device directory, the same error in two attempts. The image's libccl opens /dev/dri/by-path/. The host has that directory, but the k3d node container does not, so the Intel GPU plugin cannot pass it to the pod (smoke/H53-smoke-B2-b/hold/devdri-diag.txt).
  • Rungs (c) and (d) do not apply to that error, and the pre-registration allows no other change. So B2 does not answer, and by the pre-registered rule H53 is INCONCLUSIVE with no hold run. The finding is a gap in k3d-gpu's device exposure that affects cross-card oneCCL, not a fault in EAT.

Serving restore, with proof:

  • Both pairs were restarted with docker start at 00:36:38Z.
  • gpu-serving.sh read /health 200, /v1/models and a 16-token chat on each pair's own port (qwen38-b-hr at 00:37:40Z, qwen38-a-hr at 00:37:43Z).
  • Each pair's image, command, env, devices, mounts and ports matched its pre-stop docker inspect.
  • Through Dynamo :8100, each pair answered "OK" and its SGLang num_requests_total rose: :30000 2 to 3 at 00:37:52Z, :30001 3 to 4 at 00:38:03Z.
  • The outage was 22.7 min for qwen38-a-hr and 22.6 min for qwen38-b-hr.
  • No xe kernel message was logged in the window. No cluster, lock or process of the run is left.

Checks run locally: tests/claims_check.sh (27 files clean) and its test; tests/ledger_drift_check.py (42 files agree with 81 rows) and its test; pe_premise_check and its test; model_cards_license_test; bench/eatflow test_pairs (66), test_solo (18), test_agentbench (95), test_h4345 (46), crescent/test_holds and test_crescent; bench/apb/test_stage; shellcheck v0.11.0 at warning level on every changed or added script. All pass.

Records are in bench/eatflow/results/2026-10-20/H53/ (prehold/, window/, smoke/, runs/). The technical report now counts 32 INCONCLUSIVE and 15 PENDING, after merging main.

…and the GPU window

Amendment R1 is recorded before any H53 hold or smoke. It changes mechanics
only: no arm, model, sampling, prediction, statistic, gate, threshold, seed or
order changes, and no cap rises.

- live.sh: stage_path moves to top level. The solo path called it while it was
  defined only inside the pairs branch, so AG would have served an empty
  hostPath. The solo branch also passes LIVE_PAIRS_GUARD as --guard-s, so a
  wiring smoke runs only the warm-up.
- solo.py: alive_blocks, the blocks that can still count after each hold. The
  window stops once fewer than three can, since the verdict is then fixed at
  INCONCLUSIVE.
- prehold.sh: the window steps. E4's setup.sh under the bench lock, a model
  re-verify, front-door checks, stop and start through gpu-serving.sh (never
  rm), the AG and B2 smokes, the scored holds, per-card Prometheus records and
  a deadman that restores serving if the session dies.
…ecords

H53's window ran on crescent from 1f39488 (amendment R1). After E4's setup
and a model re-verify, both serving pairs were stopped with docker stop at
00:15Z. The wiring smokes ran, and both pairs were restored at 00:37Z.

- AG's smoke passed. agent manifest chose the GPU, and OVMS 2026.4.1 was
  ready on one card (0000:7c:00.0) and answered its warm-up.
- B2's ladder: rung (a) stops on the missing setvars.sh, as the
  pre-registration read from the image. Rung (b) gets vLLM v0.28.0 started.
  Its four XPU workers then fail in their first all_reduce: oneCCL's
  init_device_fds cannot open /dev/dri/by-path, which neither the k3d node
  container nor the pod has. Rungs (c) and (d) do not apply to that error,
  so B2 is not deliverable. By the pre-registered rule no hold runs and H53
  is INCONCLUSIVE. 0 of 8 holds were used.

Serving proof after the restore: gpu-serving.sh answered on each pair's
port, each pair's docker inspect key (image, command, env, devices, mounts,
ports) matched its pre-stop copy, and a chat through Dynamo :8100 answered
"OK" with each pair's num_requests_total rising (:30000 2 to 3, :30001 3 to
4). The technical report's counts move to 29 INCONCLUSIVE and 18 PENDING.
# Conflicts:
#	docs/intel/technical-report.md
88plug added a commit that referenced this pull request Oct 30, 2026
@88plug
88plug merged commit 9534baf into main Oct 30, 2026
79 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant