Repository navigation
batch: land #182, #236, #195, #224, #237, #226, #239, #230, #210, #233, #196, #235, #207, #213, #228, #227 - #241
Merged
Merged
Conversation
…ged tree's copy is stale #181 changed an H34-family row and merged as a ledger-only PR, which the gate lets through without a CI re-run, so console/ledger-rules.json drifted and main's console_test went red (fixed in #187). The gate now reads the tree main + head would merge to (git merge-tree) and, when its ledger-rules.json is not what its EXPERIMENTS.md generates, syncs before merging; every sync regenerates the file and commits it. The release path (hack/release_cas.sh) already regenerates it for packages. tests/gate_merge_test.sh: a stale merged tree syncs before the merge; a current one merges without a sync.
… smoke findings, plugin correction
…led container's log
…nding; VLLM_CACHE_ROOT checked against v0.24.0
…le), VLLM_CACHE_ROOT=/tmp/vllm
…preflight, endpoint check, score on exit, FIT=off for A, cap 10800 s, disclosures, smoke 6 record
…as errors H54a hold 2's B1g sample 2 stopped at 252 of 256 tokens when vLLM's engine died: the stream carried an error chunk and [DONE] but no finish_reason and no usage, and llm_load.py recorded it ok. The stub gains --die-at (that stream) and --cut-at (a connection closed mid-stream); both new checks fail against the current llm_load.py.
A request is ok only when its stream carried content, no error object and a finish_reason. vLLM ends a stream with an error chunk and [DONE] when its engine dies mid-decode, and a killed server closes the connection with no finish_reason; both were counted as short completions. Every sample now records finish_reason. tests/fake_ovms.py sets finish_reason on its last token chunk, as OVMS does (openai_completions.cpp, mapFinishReason).
… and an affinity check The runner the H57dp pre-registration asks for before its first hold. h57d.py gains the Ld-mtp-u arm (Ld-mtp without -C/--cpu-strict, every other flag unchanged), the H57dp hold orders behind 'h57d.py order ROW HOLD', and 'h57d.py score-p' for H57dp and H57de over the two holds. hold.sh takes its default order from the row, so H57d and H57w keep theirs, and refuses an unregistered H57dp hold or mode before taking the lock. The amendment adds one INCONCLUSIVE condition: each server's recorded thread affinity must show the treatment (Ld-mtp's single-CPU threads on CPUs 0,2..14 as in every H57d record, none for Ld-mtp-u). Flag defaults were read from llama.cpp b11749's source. Re-scoring the committed H57d records still reproduces verdict.json byte for byte; a test now covers it.
B1g (EAT stock vLLM CPU v0.24.0 serving Gemma 4 26B-A4B QAT) stopped mid-decode in both H54a holds and died on the 300 s execute_model RPC timeout. The records show when it stopped but not what the worker was doing. bench/h54/diag/b1g-stall.sh starts B1g's container exactly as the committed H54a render has it, with plain docker instead of k3d, under the bench lock and the hold-start rule, sends sequential 256-token greedy requests with the harness client, and on a stall records every thread of the container's processes twice and a py-spy dump of each. It is a diagnostic only: not a hold, not an arm, nothing is scored from it. Tests for the render, the thread snapshot and the stall watcher run in CI.
On crescent iaa_crypto's kernel work queue claims all 256 WQ entries of iax1, iax3 and iax5, so no user work queue fits beside it. host-setup --apply leaves those devices alone, with or without --with-kernel-wqs, and prints a shrink suggestion that nothing runs (#203). That is in policy: --apply only adds, and the shrink cuts a kernel service's own queue, which the kernel's iaa-crypto document says needs an iaa_crypto reload to take effect, with compression after a resize never tested. The note records the live sysfs read (read-only), what host-setup --apply --with-kernel-wqs does on a fake root of that state (DSA and QAT configured, no IAA command), and the consequence: H52b's IAA clause cannot hold on this host while host-setup ships as it is. No code change and no ledger amendment.
…ps its run-order place h7x.py built its block list only from the b<i>-<E>-decode.json files present. A block whose workloads ran but whose bench/energy.sh exited before llm_load left no decode record, so it was dropped: it never counted as void and every later A block paired with the wrong B block. With rates drifting 4% per A block at A = 1.12 x B, deleting block 4's records moved the verdict from SUPPORTED to INCONCLUSIVE (pairs 1.120, 1.163, 1.161, 1.160). The block list now also comes from restarts.txt, which h7x.sh writes for every block whose workloads ran; such a block with no decode record has no ok request and is void, as the pre-registration's "void blocks counted in the order" requires. A regression test removes block 4's records and expects one void A block and the same four 1.12 pairs and verdict as the void-with-records case. The ledger's plugin sentence now cites smokes 2 and 3 (smoke 1's record has no pod listing; 4 and 5 show the plugin too), the decision rule names the no-record case, and the pre-registration header records this revision; no prediction changed. h7x.sh's header gives the prefill prompt as 56 copies of one sentence, about 1850 tokens, as the ledger and the code do.
… B1g erratum The H54b holds 1-2 and H54c hold 1 records from crescent (~/k3dgpu-gemma/results/H54b, H54c and the campaign's run logs), byte-equal to the host's copies; logs over 20 kB gzipped, kubeconfigs left out. Beside the raw files, written after the holds: NOTES, the X1q and X1q-mtp runner-bug errata, and REPRODUCE-supplement.txt (host, driver, firmware, tools, GGUF sha256s). H54a hold2/B1g/ERRATUM.txt: sample 2 of load-n1.json stopped at 252 of 256 tokens when the engine died and was recorded ok; B1g completed 2 of 8 N=1 requests, not 3. The raw record is unchanged.
…e pinned pods saw all 4 cards); H54a corrections H54b and H54c rows and their run section. Every "one card" pod listed all four B70s' device nodes and llama.cpp split its layers over them, so no arm was a one-card run: both rows are INCONCLUSIVE because their arms were not delivered as pre-registered, not through predict.py's 25-40% band (its 1.261 rests on 4-card and confounded inputs). Every Gg, Hg, Gg2, Cg and Lsq number is labelled descriptive. Disclosed: the pods put VRAM and compute on the live qwen38-a-hr's two cards for about 34 minutes, which G1 did not authorise and whose impact was not measured; and this campaign stopped qwen38-b-hr itself. The X1q and X1q-mtp failures are runner bugs, with errata beside the raw records. A re-run needs an amendment that pins the device in the engine and checks isolation from inside the pod. H54a corrections (verdict unchanged): B1g completed 2 of 8 hold-2 N=1 requests, not 3; the Lg/B1g and Lg-mtp/B1g bootstrap intervals over n=2 base rates from a failing engine are struck and the ratios relabelled descriptive only; the B1g re-run probe and its harness control are named. docs/intel/technical-report.md: 25 INCONCLUSIVE, 18 PENDING; docs/intel.md gains the H54b row. The stock-arm stall is described as this harness's render of EAT's entry, cause open.
…takes a control env Run 1 found one engine thread still running while every other thread slept, and py-spy's native unwinder failed on this image. On a stall the watcher now takes 5 s of perf call graphs of each engine thread still in state R (the step run 1 did by hand) and dumps Python stacks without --native. DIAG_EXTRA_ENV adds environment to B1g's for a labelled control run, never a stock one. The first CPU sample no longer divides by a near-zero interval.
The host-state block quoted /proc/crypto's "selftest : passed" for deflate-iaa as if a self-test had run. None did. crescent's kernel has CONFIG_CRYPTO_SELFTESTS not set, and in that build the v7.0 crypto API sets CRYPTO_ALG_TESTED at registration without a test (crypto/algapi.c 278-280, 347); crypto/proc.c 49-51 then prints "passed" for any algorithm with that flag. Keep the raw read, mark it, and add a paragraph with the config read and the kernel source lines.
…uality records and the Reproduce header Follows amendment R1 on the H57dp pre-registration branch, before any H57dp hold. Every H57dp step takes LOCK=$HOME/k3dgpu-bench.lock, the lock this host's H55 runner takes, in place of the old scratchpad live.lock, and every hold runs with H57D_ROW=H57dp so H57d's spare-hold logic never reads one. The models are H57d's revisions downloaded again and checked against the committed weights.sha256, with the rebuilt drafter Q4_0 expected at 8eb21d5b7efad6e9 and 321,145,696 bytes. scripts/k3d-gpu now includes PR #180, so the hybrid vendor no longer deploys the intel-gpu-plugin H57d disclosed. score-p now reports the inherited quality records read from the committed H57d quality hold (Ld-mtp's stands for Ld-mtp-u, Od's for Od); neither row is gated on them. reproduce.sh's engine-flags line was a SyntaxError in every Python version, so all four H57d reproduce.txt files lack that block; it now prints one line per rendered arm. New tests cover the records, the bare spare-hold and smoke refusals in hold.sh, and the Reproduce header with stand-in docker, kubectl and k3d. The hold.sh and reproduce tests fail against main's scripts.
…esults files equal; 19 run files equal, 25 copied)
Run 1 started B1g's container from the committed H54a render with plain docker and stalled after 5 requests, as both H54a holds did. One worker thread spun at 1.0 CPU-s/s while every other thread slept; perf put 99.7% of its samples in the CPU attention kernel's reduce_splits, waiting on a split-completion flag in a head-dim-512 (Gemma 4 global-attention) layer. That is vllm-project/vllm#45844, a split-KV scratchpad sizing bug fixed in v0.25.0 and absent from v0.24.0, the image EAT 0d3438d ships for this model. Run 2, a labelled control with VLLM_CPU_ATTN_SPLIT_KV=0 and nothing else changed, completed 60 of 60 requests at the same speed. H54 note N2 records the runs, the cause, what EAT's stock config sets and what the H50, H51 and H54d pre-registrations do with a stock engine that dies. No row, prediction or verdict changes; the two docs that said "cause open" now name the cause.
…d order With H57D_ROW=H57dp, the hold-number check ran only when no arms were passed. Explicit arms let hold 3 through, and let hold 1 run an unregistered order and arm set that score-p would still score as hold 1. The H57dp guard now runs before the arm list is built: the hold must have a registered order, and arms given on the command line must repeat it exactly. The smoke does not set H57D_ROW and is unchanged. HoldGuards gains the two explicit-arm cases (hold 3 with Ld-mtp-u, hold 1 with Od Ld-mtp), both rc 2 with nothing written, and a check that the registered order given explicitly still gets past the guards. The amendment R1 text now says so, and its smoke paragraph lists the two 16-token requests arm.sh actually sends.
…scopes H57D_ROW to the holds
The Reproduce header prints ${G}/weights.sha256 under "file sha256:", and nothing in R2's procedure wrote that file, so every H57dp hold's header would list no file sha256. The check command now copies the committed file there once sha256sum -c passes; a mismatch copies nothing. The Row paragraph said every H57dp command sets H57D_ROW=H57dp, which the commands block does not do and does not need: only the two holds read it, the smoke writes ${G}/smoke/smoke-p under any row, and convert.sh and score-p ignore it. No row's arms, thresholds, gates, hold count, order or prediction change.
Cross-host rule 5 pins kubectl v1.35.2 in ~/k3dgpu-xhost/bin, which bench/xhost/hold.sh puts first on PATH. The node image the cross-host rows run (0.1.61) serves Kubernetes v1.37.1, two minors past the client and outside kubectl's one-minor skew policy. Crescent amendment E4 (PR #238) moves shared rule 5's workspace, ~/k3dgpu-bench/bin; it does not touch this one. K1 moves rule 5's kubectl to v1.37.2 (the newest release, 2026-10-28; k3d v5.9.0 stays), mechanics only. It covers H28x's run 2 and H7x, whose first measuring holds have not run. H25x, H27x and H34x are records and keep what they ran. A row runs all its holds on the kubectl in place at its first measuring hold. Rule 5's tools had no script in this repo. bench/xhost/tools.sh installs exactly k3d v5.9.0 and kubectl v1.37.2 into ~/k3dgpu-xhost/bin, checks both against their upstream checksum files before installing either, and prints both versions. A run against the real upstream gave "k3d: OK", "kubectl: OK", k3d v5.9.0 and kubectl v1.37.2. tests/xhost_tools_test.sh (in CI) runs it against a curl shim: exactly the two binaries at the pinned versions, four downloads, nothing else touched, a checksum mismatch on either installs nothing, no download directory left behind. Five broken copies of tools.sh (old kubectl, an install before the second check, no kubectl check, an extra binary, no cleanup) each fail it.
…lugins v0.15.0 The node image bakes the nri-plugins balloons chart v0.15.0 (component.conf, share/nri-balloons.yml, since #192), and v0.15.0 is still the newest release in the nri-plugins Helm index (2026-10-21, chart sha256 1c727a8d...). Two current-state lines still named v0.14.0: the README's `helm upgrade --install` that adopts the baked objects pinned `--version v0.14.0`, so following it would move a v0.15.0 node back to the older chart, and docs/intel.md said the image bakes balloons v0.14.0. Both now read v0.15.0. `helm template` of the chart from the repo at v0.15.0 renders the same kinds and the v0.15.0 image the baked manifest carries. The statements that describe v0.14.0's behaviour (docs/intel.md's CRD options and live run, docs/failure-modes.md's snapshot, the dated security audit, the H14 record) are left as they are.
# Conflicts: # docs/intel.md # docs/intel/technical-report.md
# Conflicts: # docs/intel/technical-report.md
# Conflicts: # docs/intel/technical-report.md
…n; its changes are already there)
…n; its changes are already there)
…aude/batch-r2 # Conflicts: # scripts/k3d-gpu
…nto claude/batch-r2
…atch-r2 # Conflicts: # tests/prefill_guard_test.sh
…nto claude/batch-r2
…ude/batch-r2 # Conflicts: # EXPERIMENTS.md
…o claude/batch-r2
…claude/batch-r2 # Conflicts: # EXPERIMENTS.md
…tch-r2 # Conflicts: # EXPERIMENTS.md # docs/intel/technical-report.md # tests/bench_test.sh
…x' into claude/batch-r2 # Conflicts: # EXPERIMENTS.md
…de/batch-r2 # Conflicts: # docs/intel/technical-report.md
…tch-r2 # Conflicts: # EXPERIMENTS.md # docs/intel.md # docs/intel/technical-report.md
…ch-r2 # Conflicts: # docs/intel/technical-report.md
Recount the technical report's verdict and row counts from EXPERIMENTS.md after the merges, name the newest run in its status line, regenerate console/ledger-rules.json for the H59 rows, and give the GGUF-conversion log hints from #182 the kubeconfig (${KHINT}) that main's kube-context test requires of user-facing hints.
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This merges reviewed PRs together so one CI run covers them. Each PR is merged with a
merge commit at the head listed here, in this order, so GitHub marks it merged when this
lands. Ledger and report conflicts are resolved keeping both sides; no prediction is changed.
d2e2dc9142ad68d7170ba5ddfa115cd1d6cb9cfd9ddb333da44e0263a5f4752fc1abe9fd9ed32a01d611f6d4b8482e5025c642c951fb937b195b7cc7a5e3a3c3d2c4a6f1e54a34b5ecda0c82f76d2e08c7d5719684b7c2b1a4ec3346856a15957e101a5b3ed1aed4ffc6872a18afa1a8131b6dbacbc8e2edbc1e601bfc0060b13e56d36d5f304dde3bd8da8cb3b0fca99bf0e9605b262752e656369a4b5a0dd57dc7bdd11448ff92ecf91f7a1b4eb707b02f1596d047c13e12562d955be196cdd80794d90a4c8eb41fe0114d7752eae3bdf13d85dc09003b8bf0e067ac585c2bb1673b5900c880ed6760b97e5dbadc914b77f2a55b8ddb154e9e4c3f6c898124d1e98d2db4e28ebccf66a5ed2f435f89dc42be3d71e37d89ce5bddf2931c2cc7b3607834a9dcc93a5c16c7304442f67919cb1f92cb644a386fd460b20b6b575fThe first merge brings in #209's merged head f0751ca, whose tree equals its squash on main
(952aa59). It changes no file; #227 and #228 carry an early #209 commit and need that
ancestry to merge cleanly. The second brings in #215's merged head cd9c032, whose tree
equals its squash on main (5cf2056). #215 was stacked on #207, so #207's content is
already on main; its merge here changes no file and only puts its head on main.
#235 and #236 are stacked on #196 and #182 (their base is that PR's branch, so GitHub
never ran CI on them); this run is their CI. #227's head is already inside #228's
branch, so its merge here adds no commit of its own.
The last commit, "batch r2: integration fixups", only recounts the technical report's
verdict and row counts from EXPERIMENTS.md, names the newest run in its status line,
regenerates console/ledger-rules.json, and points the GGUF-conversion log hints from #182
at ${KHINT} as main's kube-context test requires.