Repository navigation
ledger(H54q): INCONCLUSIVE on the Xeon 6741P; runner amendment Q2 and the run - #243
Merged
Merged
Conversation
H54q had no runner, and amendment Q1 asks for one before its first hold. This adds it. - arm.sh gains an item mode, which hold.sh turns on for row q. Each arm answers its GSM8K items 16 in flight, using H54a's chat and grader. Every finished request is written as it lands. No request starts after an item stop 90 s before the hold's deadline. An item's first error re-asks it once; a second error counts as wrong. - When the server stops answering, the arm is failed and the run of errors is discarded. - OVQ, OVB and Lg apply the manifests H54a's scored runs applied, byte for byte. R and Vi derive from H54a's recorded B1g render. - qcampaign.sh plans each of the three holds from the records so far. - h54.py score-q pairs items 0-199 and bootstraps OVQ - R and OVB - OVQ (10,000 resamples, seed 1). - The tests cover the renders, the record rules, the plan, the client against fakes and a local stub server, and the verdicts.
…ecord around the holds Neither step is part of H54q's protocol or score. verify hashes every weight file the arms load against the pre-registration's sha256 list and OVQ's manifest, under the bench lock. frontdoor records that both SGLang serving pairs still answer through Dynamo :8100, with each pair's request counter rising.
88plug
added a commit
that referenced
this pull request
Oct 29, 2026
… hold The amendment's facts are brought up to date; no pin, checksum, check or render changes, and every pin it names was read again on 2026-10-29 at 19:50Z (kubectl v1.37.2, helm v4.3.1 and uv 0.13.6 are still the newest, with the same sha256; k3d v5.9.0, EAT release-1.1.0, EAB v1.0.0 and the LiteLLM index digest 73cbb215 are unchanged). - Item 2: H52b came first. Its one hold (18:32-18:43Z, PR #232) ran before setup.sh, on shared rule 5's k3d v5.9.0, kubectl v1.35.2 and helm v4.1.1, as its records name them. - Item 4: H59 and H60 have merged. H60's Reproduce block runs it from the cross-host workspace ~/k3dgpu-xhost/bin, which amendment K1 covers and setup.sh does not touch, so E4 does not cover it. H59 has no runner yet; its runner amendment names its tools. - Item 5: H54q's runner amendment Q2 was pushed at 19:34Z (PR #243). It keeps H54a's tools in ~/k3dgpu-gemma (kubectl v1.35.2) and renders no arm with helm, so E4 does not touch H54q.
This was referenced Oct 29, 2026
…and the checks Three holds on crescent under amendments Q1 and Q2, plus one discarded wiring smoke. Each arm's turn has items.jsonl (one row per finished request), q.json, status.json, deploy.yaml, server.log and its readiness files. Each hold has reproduce.txt, pre.txt, post.txt, serving.jsonl and hold.log. The campaign has plan-hold1-3.txt, not-run.txt and score-q.json. The weight check and the before and after front-door records are included. Kubeconfigs are left out, and logs over 20 kB are gzipped.
…s the bf16 quality, the base-model IR is not worse H54q ran to its pre-registered protocol (amendment Q1, runner amendment Q2) in 3 of its 3 holds. R, OVQ, OVB and Lg answered all of GSM8K items 0-199, with no errored item. Correct of 200: R (bf16 QAT, vLLM CPU v0.32.0) 194, OVQ 191, OVB 192, Lg 193. - OVQ - R is -0.015 (95% CI -0.035 to 0.000). Its lower bound clears the -0.05 margin. - OVB - OVQ is +0.005 (CI -0.015 to +0.025). It is neither below 0 nor above it. - Only one SUPPORTED bound holds and no REFUTED bound does, so the row reads INCONCLUSIVE. That is the verdict it predicted most likely. - Vi was not deliverable: vLLM v0.32.0 cannot load the AutoRound MoE checkpoint. This commit adds the row's verdict, a falsification-log entry and the run section with its Reproduce header. The technical report now counts 29 INCONCLUSIVE and 19 PENDING and gains an H54q row. docs/intel.md's H54q row is updated, and the README no longer says H54q has yet to measure the QAT conversion.
# Conflicts: # docs/intel/technical-report.md
88plug
marked this pull request as ready for review
October 29, 2026 21:26
88plug
added a commit
that referenced
this pull request
Oct 29, 2026
… hold The amendment's facts are brought up to date; no pin, checksum, check or render changes, and every pin it names was read again on 2026-10-29 at 19:50Z (kubectl v1.37.2, helm v4.3.1 and uv 0.13.6 are still the newest, with the same sha256; k3d v5.9.0, EAT release-1.1.0, EAB v1.0.0 and the LiteLLM index digest 73cbb215 are unchanged). - Item 2: H52b came first. Its one hold (18:32-18:43Z, PR #232) ran before setup.sh, on shared rule 5's k3d v5.9.0, kubectl v1.35.2 and helm v4.1.1, as its records name them. - Item 4: H59 and H60 have merged. H60's Reproduce block runs it from the cross-host workspace ~/k3dgpu-xhost/bin, which amendment K1 covers and setup.sh does not touch, so E4 does not cover it. H59 has no runner yet; its runner amendment names its tools. - Item 5: H54q's runner amendment Q2 was pushed at 19:34Z (PR #243). It keeps H54a's tools in ~/k3dgpu-gemma (kubectl v1.35.2) and renders no arm with helm, so E4 does not touch H54q.
… 200 items in H54q Re-grading every arm's 200 items from the raw records (h54.q_state over hold1/R, hold1/OVQ, hold2/OVB, hold2/Lg and H54a's imported items 0-49) finds that items 85, 119 and 187 defeated all four arms. Item 182 did not: OVB answered it (hold2/OVB/items.jsonl, idx 182, ok true, gold 23). The row's result text and the run section said R, OVQ, OVB and Lg answered all 200 items in holds 1 and 2. Only R did; OVQ, OVB and Lg answered items 50-199 there, and their items 0-49 are H54a's records (Q1 item 2), as the method column, the What ran table and the Disclosed bullet already state. No count, test, CI, prediction or verdict changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
H54q is INCONCLUSIVE, the verdict its pre-registration rated most likely (0.65).
On crescent's Xeon 6741P, over GSM8K items 0-199 with Gemma 4 26B-A4B:
Result (
h54.py score-q; paired bootstrap over items, 10,000 resamples, seed 1)What ran
Amendment Q2, the runner. Q1 required a runner amendment before the first hold. Q2 was pushed at 19:34:33Z (11e076d), before any smoke or hold. As with H50's E3 and H51's S1, it was not merged when the holds ran; the run section discloses this.
arm.shgained an item mode. Each arm answers its GSM8K items 16 in flight, with H54a's chat and grader unchanged. Every finished request is written as it lands, and no request starts after an item stop 90 s before the hold's deadline.h54.py score-qassembles items 0-199: H54a's records for items 0-49 of OVQ, OVB and Lg, and this run's records for the rest.Before the holds, at 85e7726 (the runner plus
qcheck.sh):The holds (
qcampaign.sh), 3 of the 3 Q1 allows:Crescent after the run
num_requests_totalrose:Changes
bench/h54/:h54.py: theq-*commands andscore-q.arm.sh: the item mode.hold.sh: rowqand the smoke rowsmokeq.reproduce.sh: the H54q manifests' sha256.qcampaign.shandqcheck.sh, both new.q-smoke.jsonl: the smoke's two non-GSM8K questions.test_h54.py: H54q tests, already in CI.bench/h54/results/2026-10-29/H54q/: every record from the three holds, the smoke and the checks.EXPERIMENTS.md:docs/intel/technical-report.md: the counts now read 29 INCONCLUSIVE and 18 PENDING (after H52b, merged from main), and there is an H54q INCONCLUSIVE row.docs/intel.md: the H54q row is updated.README.md: the QAT-conversion paragraph now says H54q's run did not promote the conversion. The optimizer still promotes it only on a SUPPORTED H54q.Checks run locally
bench/h54/test_h54.py: OK, 54 tests. Repeated runs were stable.bench/h54/diag/test_b1g_diag.py: OK.bench/h54gpu/test_h54gpu.py: OK.bench/eatflow/crescent/test_holds.py: OK.bench/eatflow/crescent/test_crescent.py: OK.tests/bench_test.sh: 71 passed, 0 failed.tests/claims_check.sh: 26 files clean.tests/claims_check_test.sh: 81 passed.python3 tests/ledger_drift_check.py: 41 files agree with 81 ledger rows.tests/ledger_drift_check_test.py: OK.tests/pe_premise_check.sh: ok.tests/pe_premise_check_test.sh: 16 passed.tests/model_cards_license_test.py: OK.tests/ci_wiring_test.sh: 69 passed.tests/docs_test.shwith mkdocs 1.6.1: 63 passed.console/ledger-rules.jsonregenerates byte-identical.bench/h54/*.sh.score-q.jsonbyte for byte.