Skip to content

ledger(H54q): INCONCLUSIVE on the Xeon 6741P; runner amendment Q2 and the run - #243

Merged
88plug merged 6 commits into
mainfrom
claude/h54q-run
Oct 30, 2026
Merged

88plug merged 6 commits into
mainfrom
claude/h54q-run

Conversation

@88plug

@88plug 88plug commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

H54q is INCONCLUSIVE, the verdict its pre-registration rated most likely (0.65).

On crescent's Xeon 6741P, over GSM8K items 0-199 with Gemma 4 26B-A4B:

  • The QAT-matched OpenVINO IR (OVQ: INT4_SYM, group 32, built from Google's QAT bf16 weights) stayed inside the 5-point non-inferiority margin of the bf16 QAT reference (R).
  • OpenVINO's published base-model IR (OVB) did not score below OVQ. The second half of the claim therefore does not hold, and no REFUTED bound is crossed.

Result (h54.py score-q; paired bootstrap over items, 10,000 resamples, seed 1)

Arm Correct of 200
R: vLLM CPU v0.32.0, QAT bf16 194 (0.970)
OVQ: OVMS 2026.4.1 191 (0.955)
OVB: OVMS 2026.4.1 192 (0.960)
Lg: llama.cpp b11749, QAT Q4_0 GGUF 193 (0.965)
Vi: vLLM v0.32.0, Intel AutoRound not deliverable
  • OVQ - R: -0.015, 95% CI -0.035 to 0.000. The lower bound clears -0.05, so the SUPPORTED half holds.
  • OVB - OVQ: +0.005, CI -0.015 to +0.025. It is neither below 0 nor above 0, so the SUPPORTED half fails and the REFUTED bound is not met.
  • Not gated:
    • Lg - R: -0.005 (CI -0.025 to +0.015).
    • OVB - R: -0.010 (CI -0.035 to +0.015).
  • Predictions:
    • R in [0.85, 0.96]: missed. R read 0.970, above the range.
    • OVQ - R, Lg - R and OVB - R: each inside its range.
    • "OVB < OVQ resolves": did not resolve, as the 0.8 side predicted.
  • Every build scored 95.5-97%. Only 5 items differ between OVB and OVQ, too few to separate them, so the falsification log closes this item set for these builds.

What ran

Amendment Q2, the runner. Q1 required a runner amendment before the first hold. Q2 was pushed at 19:34:33Z (11e076d), before any smoke or hold. As with H50's E3 and H51's S1, it was not merged when the holds ran; the run section discloses this.

  • arm.sh gained an item mode. Each arm answers its GSM8K items 16 in flight, with H54a's chat and grader unchanged. Every finished request is written as it lands, and no request starts after an item stop 90 s before the hold's deadline.
  • An item's first error re-asks it once; a second error counts as wrong.
  • A server that stops answering fails its arm, and that run of errors is discarded.
  • OVQ, OVB and Lg apply the exact manifests H54a's scored runs applied. R and Vi derive from H54a's recorded B1g render. R's manifest is byte-identical to the Vq manifest H54a's second smoke booted.
  • h54.py score-q assembles items 0-199: H54a's records for items 0-49 of OVQ, OVB and Lg, and this run's records for the rest.

Before the holds, at 85e7726 (the runner plus qcheck.sh):

  • Every weight file's sha256 matched the pre-registration's list, read under the bench lock.
  • Both serving pairs answered through Dynamo :8100, each with its request counter rising.
  • A discarded wiring smoke ran: R, OVQ and Lg were ready and answered two non-GSM8K questions. No GSM8K item was asked.

The holds (qcampaign.sh), 3 of the 3 Q1 allows:

Hold Time (UTC) What it did
1 19:46-20:20 R answered items 0-199 in 542 s. OVQ answered items 50-199 in 1,253 s.
2 20:22-20:59 OVB answered items 50-199 in 1,098 s. Lg answered items 50-199 in 672 s.
3 20:59-21:15 Vi only, not deliverable. vLLM v0.32.0 stops on the AutoRound checkpoint with "has no parameter 'w2_qweight'", as in H54a's smoke.
  • No item errored or was cut, and no server stopped answering.
  • aiperf was idle in all 521 sampler windows.

Crescent after the run

  • No k3d cluster is left, no lock is held, and no process of this run is running.
  • Both serving pairs answered a chat through the Dynamo :8100 front door, and each pair's num_requests_total rose:
    • 19:37:01Z, before: :30000 157 to 158, :30001 52 to 53.
    • 21:15:57Z, after: :30000 158 to 159, :30001 54 to 55.
  • H54q used no GPU and stopped no container.

Changes

  • bench/h54/:
    • h54.py: the q-* commands and score-q.
    • arm.sh: the item mode.
    • hold.sh: row q and the smoke row smokeq.
    • reproduce.sh: the H54q manifests' sha256.
    • qcampaign.sh and qcheck.sh, both new.
    • q-smoke.jsonl: the smoke's two non-GSM8K questions.
    • test_h54.py: H54q tests, already in CI.
  • bench/h54/results/2026-10-29/H54q/: every record from the three holds, the smoke and the checks.
  • EXPERIMENTS.md:
    • amendment Q2;
    • the H54q row's verdict and result;
    • a falsification-log entry;
    • the "H54q run" section, with its Reproduce header.
  • docs/intel/technical-report.md: the counts now read 29 INCONCLUSIVE and 18 PENDING (after H52b, merged from main), and there is an H54q INCONCLUSIVE row.
  • docs/intel.md: the H54q row is updated.
  • README.md: the QAT-conversion paragraph now says H54q's run did not promote the conversion. The optimizer still promotes it only on a SUPPORTED H54q.

Checks run locally

  • bench/h54/test_h54.py: OK, 54 tests. Repeated runs were stable.
  • bench/h54/diag/test_b1g_diag.py: OK.
  • bench/h54gpu/test_h54gpu.py: OK.
  • bench/eatflow/crescent/test_holds.py: OK.
  • bench/eatflow/crescent/test_crescent.py: OK.
  • tests/bench_test.sh: 71 passed, 0 failed.
  • tests/claims_check.sh: 26 files clean. tests/claims_check_test.sh: 81 passed.
  • python3 tests/ledger_drift_check.py: 41 files agree with 81 ledger rows. tests/ledger_drift_check_test.py: OK.
  • tests/pe_premise_check.sh: ok. tests/pe_premise_check_test.sh: 16 passed.
  • tests/model_cards_license_test.py: OK.
  • tests/ci_wiring_test.sh: 69 passed.
  • tests/docs_test.sh with mkdocs 1.6.1: 63 passed.
  • console/ledger-rules.json regenerates byte-identical.
  • shellcheck v0.11.0 at warning level is clean on bench/h54/*.sh.
  • The scorer, run on the committed records, reproduces crescent's score-q.json byte for byte.

H54q had no runner, and amendment Q1 asks for one before its first hold. This adds it.

- arm.sh gains an item mode, which hold.sh turns on for row q. Each arm answers its GSM8K items 16 in flight, using H54a's chat and grader. Every finished request is written as it lands. No request starts after an item stop 90 s before the hold's deadline. An item's first error re-asks it once; a second error counts as wrong.
- When the server stops answering, the arm is failed and the run of errors is discarded.
- OVQ, OVB and Lg apply the manifests H54a's scored runs applied, byte for byte. R and Vi derive from H54a's recorded B1g render.
- qcampaign.sh plans each of the three holds from the records so far.
- h54.py score-q pairs items 0-199 and bootstraps OVQ - R and OVB - OVQ (10,000 resamples, seed 1).
- The tests cover the renders, the record rules, the plan, the client against fakes and a local stub server, and the verdicts.
…ecord around the holds

Neither step is part of H54q's protocol or score. verify hashes every weight file the arms load against the pre-registration's sha256 list and OVQ's manifest, under the bench lock. frontdoor records that both SGLang serving pairs still answer through Dynamo :8100, with each pair's request counter rising.
88plug added a commit that referenced this pull request Oct 29, 2026
… hold

The amendment's facts are brought up to date; no pin, checksum, check or
render changes, and every pin it names was read again on 2026-10-29 at
19:50Z (kubectl v1.37.2, helm v4.3.1 and uv 0.13.6 are still the newest,
with the same sha256; k3d v5.9.0, EAT release-1.1.0, EAB v1.0.0 and the
LiteLLM index digest 73cbb215 are unchanged).

- Item 2: H52b came first. Its one hold (18:32-18:43Z, PR #232) ran
  before setup.sh, on shared rule 5's k3d v5.9.0, kubectl v1.35.2 and
  helm v4.1.1, as its records name them.
- Item 4: H59 and H60 have merged. H60's Reproduce block runs it from
  the cross-host workspace ~/k3dgpu-xhost/bin, which amendment K1 covers
  and setup.sh does not touch, so E4 does not cover it. H59 has no
  runner yet; its runner amendment names its tools.
- Item 5: H54q's runner amendment Q2 was pushed at 19:34Z (PR #243). It
  keeps H54a's tools in ~/k3dgpu-gemma (kubectl v1.35.2) and renders no
  arm with helm, so E4 does not touch H54q.
…and the checks

Three holds on crescent under amendments Q1 and Q2, plus one discarded wiring smoke. Each arm's turn has items.jsonl (one row per finished request), q.json, status.json, deploy.yaml, server.log and its readiness files. Each hold has reproduce.txt, pre.txt, post.txt, serving.jsonl and hold.log. The campaign has plan-hold1-3.txt, not-run.txt and score-q.json. The weight check and the before and after front-door records are included. Kubeconfigs are left out, and logs over 20 kB are gzipped.
…s the bf16 quality, the base-model IR is not worse

H54q ran to its pre-registered protocol (amendment Q1, runner amendment Q2) in 3 of its 3 holds. R, OVQ, OVB and Lg answered all of GSM8K items 0-199, with no errored item. Correct of 200: R (bf16 QAT, vLLM CPU v0.32.0) 194, OVQ 191, OVB 192, Lg 193.

- OVQ - R is -0.015 (95% CI -0.035 to 0.000). Its lower bound clears the -0.05 margin.
- OVB - OVQ is +0.005 (CI -0.015 to +0.025). It is neither below 0 nor above it.
- Only one SUPPORTED bound holds and no REFUTED bound does, so the row reads INCONCLUSIVE. That is the verdict it predicted most likely.
- Vi was not deliverable: vLLM v0.32.0 cannot load the AutoRound MoE checkpoint.

This commit adds the row's verdict, a falsification-log entry and the run section with its Reproduce header. The technical report now counts 29 INCONCLUSIVE and 19 PENDING and gains an H54q row. docs/intel.md's H54q row is updated, and the README no longer says H54q has yet to measure the QAT conversion.
# Conflicts:
#	docs/intel/technical-report.md
@88plug 88plug changed the title ledger(H54q): the runner (amendment Q2) and the run ledger(H54q): INCONCLUSIVE on the Xeon 6741P; runner amendment Q2 and the run Oct 29, 2026
@88plug
88plug marked this pull request as ready for review October 29, 2026 21:26
88plug added a commit that referenced this pull request Oct 29, 2026
… hold

The amendment's facts are brought up to date; no pin, checksum, check or
render changes, and every pin it names was read again on 2026-10-29 at
19:50Z (kubectl v1.37.2, helm v4.3.1 and uv 0.13.6 are still the newest,
with the same sha256; k3d v5.9.0, EAT release-1.1.0, EAB v1.0.0 and the
LiteLLM index digest 73cbb215 are unchanged).

- Item 2: H52b came first. Its one hold (18:32-18:43Z, PR #232) ran
  before setup.sh, on shared rule 5's k3d v5.9.0, kubectl v1.35.2 and
  helm v4.1.1, as its records name them.
- Item 4: H59 and H60 have merged. H60's Reproduce block runs it from
  the cross-host workspace ~/k3dgpu-xhost/bin, which amendment K1 covers
  and setup.sh does not touch, so E4 does not cover it. H59 has no
  runner yet; its runner amendment names its tools.
- Item 5: H54q's runner amendment Q2 was pushed at 19:34Z (PR #243). It
  keeps H54a's tools in ~/k3dgpu-gemma (kubectl v1.35.2) and renders no
  arm with helm, so E4 does not touch H54q.
… 200 items in H54q

Re-grading every arm's 200 items from the raw records (h54.q_state over
hold1/R, hold1/OVQ, hold2/OVB, hold2/Lg and H54a's imported items 0-49)
finds that items 85, 119 and 187 defeated all four arms. Item 182 did not:
OVB answered it (hold2/OVB/items.jsonl, idx 182, ok true, gold 23).

The row's result text and the run section said R, OVQ, OVB and Lg answered
all 200 items in holds 1 and 2. Only R did; OVQ, OVB and Lg answered items
50-199 there, and their items 0-49 are H54a's records (Q1 item 2), as the
method column, the What ran table and the Disclosed bullet already state.

No count, test, CI, prediction or verdict changes.
@88plug
88plug merged commit 7c6d3fa into main Oct 30, 2026
78 of 79 checks passed
88plug added a commit that referenced this pull request Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant