[juliankang4]

기본 prepare가 체크포인트의 BF16 가중치를 Q8_0으로 바꾸지 않고 저장된 형식 그대로 쓰게 하고, Q8_0 재작성은 BLOOMERY_PREPARE_Q8 레버로 남깁니다.

Part of #565.

What and why

prepare now writes a BF16 matrix of a checkpoint as BF16. NVFP4, MXFP8 and FP8 tensors are kept as stored, as before. Until now prepare rewrote the matrices that the architecture map marks as Q8_0 unless --keep-bf16 was given. The rewrite is now opt-in through the lever BLOOMERY_PREPARE_Q8=1.

The oracle holds separate reference sets for the BF16 and the Q8_0 dense form, so the form a split holds decides which set the NVFP4 gate must judge it by. The BF16 dense path on the card already exists. This PR makes it the default output of prepare and makes the gate judge a split by the dense form it holds.

The converter has no support for a BF16 or F32 routed expert stack. Both modes (default and BLOOMERY_PREPARE_Q8=1) refuse such a stack before any shard is written, and the error names the HF and GGUF tensor names and the stored format. An F32 dense matrix with the Q8_0 role is refused in the default mode with its tensor names and BLOOMERY_PREPARE_Q8=1, because the card program reads a dense projection only as Q8_0 or BF16. With the lever set, the matrix is written as Q8_0.

Q8_0 is not the same numerically as the stored BF16. In the CPU oracle forward, rounding the BF16 projections to Q8_0 moves the last position by KL 4.288e-02 on the batch set and by KL 4.181e-02 at position 3000 of the D3K set. The lever's help text names these figures.

• crates/model-hf/src/bin/prepare.rs: plan_tensors builds the plan, the map and the declarations in one place. A stored BF16 matrix whose map role is Q8_0 keeps its format unless the lever is set. The refusals above happen here. --keep-bf16 is still accepted; alone it equals the default, and next to the lever it is refused by name.
• crates/levers/src/registry.rs, crates/levers/src/lib.rs: the BLOOMERY_PREPARE_Q8 row (Class M, flag, default 0, parsed, load) and the prepare_q8() accessor. The Draft38Off::Borrowed text now says "not a form the MTP draft reads", which matches what mtp_borrows accepts (embedding Q8_0 or BF16, head Q8_0, Q6_K or BF16).
• crates/gpu-gates/src/bin/gate_qwen4exp_nvfp4.rs: dense_of reads the type of output.weight and picks the judged oracle sets. A BF16 head is judged by the _bf16 pair, a Q8_0 head by the Q8_0 pair, whoever wrote the split. Tensors of the next-token layer that the load does not use are left out of that judgement. A split that mixes BF16 and Q8_0 dense tensors, or has a head of another type or none, is refused by tensor name. The indexer q and k projections are exempt from the mixed check, because the oracle holds them BF16 beside either form and prepare keeps them BF16. This only selects a reference set: the model loader refuses Q8_0 indexer projections, so a split with them does not load. The judged-set line says when the indexer includes Q8_0. The gate takes its model path from the existing BLOOMERY_REF_MODEL, resolved once for the dimensions and the load.
• justfile: gate-gpu-qwen4exp-nvfp4 keeps the same default BLOOMERY_REF_MODEL as before this PR and runs the bin's unit test first, like gate-gpu-bf16. A split written by the new default prepare is selected by setting BLOOMERY_REF_MODEL.
• crates/model-hf/Cargo.toml, Cargo.lock: a dependency on bloomery-levers. Two doc comments in crates/refset/src/arch/qwen4exp/nvfp4.rs and crates/model/src/arch/qwen35moe/mtp.rs follow the new default.

Release notes: Added: prepare keeps a checkpoint's BF16 matrices in their stored format by default; the Q8_0 rewrite is available through BLOOMERY_PREPARE_Q8=1, and the NVFP4 gate judges a split by its dense form.

Code shape

• No second copy. The lever has one registry row. The list of tensors that can be rewritten stays in the model-hf arch map, so the lever and --keep-bf16 act on the same list. The choice of judged sets is one function, dense_of, and the gate reuses the existing model path override and the load's own tensor roles.
• Model code holds only architecture facts. The format choice lives in plan_tensors. The map's roles are unchanged.
• Dead code is gone. The judging decisions that were spread over dims and run in the gate moved into dense_of.
• No silent failure. A malformed lever value, --keep-bf16 next to the lever, an F32 matrix in the default mode, a BF16 or F32 routed stack in either mode, a split without a head, a head of a third type and a mixed split are each refused by name.
• Comments state what is true now.
• No relaxed gate. No band, tolerance or refusal was removed or loosened.
• cargo fmt is applied to the changed files.

Duplication found or removed: none.

What I ran

The PR head is 86fb17366bc9a90181aa5ddee8a214a89193aeae on upstream main 0f7b51c6ebd2214b6393c841d2ae108354984de7. The unit tests and the gate ran on an earlier revision of this branch, which sat on main 938167b2997b48b60593c27956efd36b352a8fad. Since then the diff changed only in documentation comments (the gate module and dense_of docs, one paragraph in prepare.rs, the help sentence of BLOOMERY_PREPARE_Q8) and a test fixture in prepare.rs, and the branch was rebased onto main with its #603 (serve), #604 (tools/ref scripts) and #606 (the fixture budget check). None of these touches anything the gate builds or reads. With the prepare source of the PR head, prepare on the same checkpoint writes the four shards of the split the gate judged byte for byte (see the check below), so the gate result carries to this head.

Unit tests of prepare. The test covers the BF16 default, the refusal of F32 and BF16 expert stacks in both modes (including a stack of one expert), the F32 matrix in the default mode and the Q8_0 opt-in:

test tests::a_bf16_matrix_is_written_as_stored_unless_q8_is_asked ... ok
test result: ok. 4 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out

The gate's test of the judged sets, including the unused next-token layer tensors and the mixed split:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Each of these checks was first run against the code with its defect put back, and it caught the defect. A stack of one expert was let through in both modes when the stack test counted source tensors instead of the tensor's rank:

Bf16 expert stacks must be refused for q8=false, entries=3
Bf16 expert stacks must be refused for q8=true, entries=3

With the default forced to Q8_0, the BF16 matrix came out as Q8_0:

assertion `left == right` failed: blk.0.attn_gate.weight
  left: (Q8_0, Q8_0)
 right: (BF16, Q8_0)

With the mixed-format refusal deleted from dense_of, the gate's test took a mixed split as Q8_0:

called `Result::unwrap_err()` on an `Ok` value: Dense { ty: Q8_0, q8_indexer: false,

GPU evidence

Card, from nvidia-smi on the DGX Spark that ran the gates:

| NVIDIA-SMI 595.84                 Driver Version: 595.84         CUDA Version: 13.3     |
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |

The gate command, run once per split. BLOOMERY_REF_MODEL was set to the first shard of each split, so the shown default path applies to the second run only; no source file was edited between the runs.

BLOOMERY_REF_MODEL="${BLOOMERY_REF_MODEL:-/models/Qwen3.8-Flash-Next-NVFP4/Qwen3.8-Flash-Next-NVFP4-00001-of-00004.gguf}" BLOOMERY_MODEL=qwen4exp ./tools/box.sh 'bash tools/ref/real-only.sh gate-gpu-qwen4exp-nvfp4 && bash tools/gate.sh --oxide -p bloomery-gpu-gates --release --features gpu --bin gate_qwen4exp_nvfp4 && cargo oxide build --arch sm_121 -- -p bloomery-gpu-gates --features gpu --release --bin gate_qwen4exp_nvfp4 && BLOOMERY_GATE_CARD=${BLOOMERY_GATE_CARD:-any} bash tools/gpu-gate.sh gate_qwen4exp_nvfp4'

On the split with BF16 dense parts as stored, the gate judged by the BF16 reference sets:

model /models/Qwen3.8-Flash-Next-NVFP4-shipped/Qwen3.8-Flash-Next-NVFP4-00001-of-00004.gguf
dense projections BF16 as stored: judged by hf-qwen4exp-nvfp4/hf_qwen4exp_nvfp4_bf16 and hf-qwen4exp-nvfp4/hf_qwen4exp_nvfp4_d3k_bf16; indexer BF16, the oracle's BF16
(l) plan card=GB10 ctx_max=3072 ubatch=3072 host_experts=0 card_experts=24576 layers short of 512 card experts: [] PASS
(p) the batch set by one pass: last argmax ours=11751 oracle=11751 logits_rel=8.243e-2 (printed) PASS
(g) the batch set by one ubatch: last argmax ours=11751 oracle=11751 logits_rel=5.579e-2 (printed) PASS
(t) hf-qwen4exp-nvfp4/hf_qwen4exp_nvfp4_d3k_bf16: position 3000 after 3000 ids by ubatches
    argmax ours=423 oracle=423 (its runner-up 11222, margin 0.9281, our distance at the two 0.3040); logits_rel 1.112e-1; worst l_out_rel 1.839e-1 (printed) PASS
GATE-RC stored 0 (just gate-gpu-qwen4exp-nvfp4)

On the Q8_0 split, the gate judged by the Q8_0 sets:

model /models/Qwen3.8-Flash-Next-NVFP4/Qwen3.8-Flash-Next-NVFP4-00001-of-00004.gguf
dense projections Q8_0: judged by hf-qwen4exp-nvfp4/hf_qwen4exp_nvfp4 and hf-qwen4exp-nvfp4/hf_qwen4exp_nvfp4_d3k; indexer BF16, the oracle's BF16
(l) plan card=GB10 ctx_max=3072 ubatch=3072 host_experts=0 card_experts=24576 layers short of 512 card experts: [] PASS
(p) the batch set by one pass: last argmax ours=11751 oracle=11751 logits_rel=8.401e-2 (printed) PASS
(g) the batch set by one ubatch: last argmax ours=11751 oracle=11751 logits_rel=9.377e-2 (printed) PASS
(t) hf-qwen4exp-nvfp4/hf_qwen4exp_nvfp4_d3k: position 3000 after 3000 ids by ubatches
    argmax ours=423 oracle=423 (its runner-up 11222, margin 1.6376, our distance at the two 0.2279); logits_rel 1.070e-1; worst l_out_rel 1.784e-1 (printed) PASS
GATE-RC q8 0 (just gate-gpu-qwen4exp-nvfp4)

Comparison against the reference sets, taken from the oracle dump of the checkpoint:

Reference	Result
just gate-gpu-qwen4exp-nvfp4 on the split prepare now writes (BF16 dense parts as stored) against the BF16 reference sets	every clause PASS, last argmax 11751 equal to the reference, exit 0

The split prepare writes by default (BF16 dense parts kept BF16) passes the real Qwen3.8 NVFP4 gate against its own BF16 reference sets; a Q8_0 split written by the earlier prepare passes against the Q8_0 sets.

At this head, prepare on checkpoint fc694b54 wrote four shards, and each matches the SHA256SUMS of the stored-format split the gate judged:

PREPARE-RC 0
Qwen3.8-Flash-Next-NVFP4-00001-of-00004.gguf: OK
Qwen3.8-Flash-Next-NVFP4-00002-of-00004.gguf: OK
Qwen3.8-Flash-Next-NVFP4-00003-of-00004.gguf: OK
Qwen3.8-Flash-Next-NVFP4-00004-of-00004.gguf: OK
SAME-BYTES-RC 0

To reproduce, from the repository root with CUDA configured:

cargo test -p bloomery-model-hf --bin prepare
cargo run --release -p bloomery-model-hf --bin prepare -- <checkpoint folder> <out prefix>
BLOOMERY_REF_MODEL=<first shard prepare wrote> just gate-gpu-qwen4exp-nvfp4

This PR changes no kernel and makes no speed claim. BF16 dense reads more bytes per token than Q8_0, so decode is expected to be slower than on a Q8_0 split.

What I ran into

The following does not block this PR. The gate's selection test is a bin test, so it runs only where cargo oxide is available. The "Q8_0 split from another converter" case, the mixed split and the unused next-token layer case in that test are synthetic lists of names, types and roles, not a real published file. The F32 and BF16 stack refusals are tested with synthetic tensor headers, not with a published checkpoint that stores them. The Q8_0 split in the second gate run was written by the previous default prepare, not by the new lever, so the run does not show the lever's output byte for byte. The levers unit tests were not rerun at this head; the registry change since their last run is a help sentence. The comments of tools/ref/qwen4exp-nvfp4-oracle.py still describe the BF16 projections as the ones prepare writes as Q8_0. The writer is pinned by its md5 together with the sets it wrote, so a comment edit there needs the sets written again; the refset module doc (crates/refset/src/arch/qwen4exp/nvfp4.rs) states the current default.

Hardware

A DGX Spark (GB10) for the gates. The nvidia-smi lines of the card are under GPU evidence. A separate CPU host ran the prepare unit tests.

AI help

AI 코딩 에이전트가 코드를 작성한 후 설명을 썼고, 테스트를 마쳤습니다. diff 리뷰와 PR 초안 리뷰는 제가 직접 했습니다.

[github-actions]

Verdict: CLEAN: No behavior or quality defect survived the format-policy and reference-selection checks.

Checked

Duplicates: 7 new items searched; no duplicate owner found. Dryer reported 0 pairs; gpu-gates was checked by Grep.
Complexity: plan_tensors is new at 20: placement/mapping, then declarations and format refusals. Gate run rose 19→20 through format reporting. Both are outside the inference hot path.
Silent failure: 4 filter/default sites followed to their consumers; none discards an error. Unused tensors and tensors loaded as widened values are intentionally excluded from dense-form selection.
Tests: Static mutation checks: forcing Q8_0 or dropping the single-expert/F32 refusals at crates/model-hf/src/bin/prepare.rs:271 breaks the format assertions. Swapping reference pairs or dropping the mixed check at crates/gpu-gates/src/bin/gate_qwen4exp_nvfp4.rs:348 breaks the selection assertions. Copying F32 bytes through the Q8_0 branch at crates/hf/src/modelopt.rs:615 breaks the byte assertions. Inverting prepare_q8 at crates/levers/src/lib.rs:1190 breaks accessor assertions; restoring the old message at crates/levers/src/lib.rs:2082 breaks the display assertion. Removing prepare's size check (line 95) or conflict check (line 89) breaks the corresponding error assertions. Overwriting an existing draft refusal in paged_draft breaks its preservation assertion.
Gates: No threshold or pin moved. Both existing oracle pairs retain their checks; the PR includes FAIL-first output.
Adversarial: NaN reaches the finite-value rejection; one-expert stacks retain rank three and are refused before writing; unused next-token tensors cannot change the selected sets. Additional clean check: Q8_0 indexer projections are rejected by the loader's F32/BF16 shape check.
Escape catalog: C1–C4 have no matching changed pattern.

Notes

Tests were inspected, not executed in this review. The PR supplies GB10 gate output from an earlier revision and reports byte-identical regenerated shards at head.

_{Card check: gpu-evidence. This pull request changes crates/gpu-gates/src/bin/gate_qwen4exp_nvfp4.rs and carries a gate run on the author's card, which counts as verified.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[juliankang4]

The review is done and the contribution check passed, so I'm merging this.