Skip to content

th#1803: the consumer half of the layout gate — a decoder that DECLARES hf.fp8-blockwise@1 - #662

Merged
PaulFidika merged 1 commit into
masterfrom
1803-blockwise-loader
Aug 11, 2026
Merged

th#1803: the consumer half of the layout gate — a decoder that DECLARES hf.fp8-blockwise@1#662
PaulFidika merged 1 commit into
masterfrom
1803-blockwise-loader

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

tensorhub 226070fd (th#1803, PR #1046) made the tensor-layout contract enforceable at rebind — a binding whose artifact bytes no decoder in the target image declares is refused with a typed 422, no pod bought. It refused the H3 fp8 conditioner rebind correctly: nothing in any image declared hf.fp8-blockwise@1, the contract tensorhub transcribed from that artifact's own 165,264 header bytes.

This is the consumer half: the declaration, and the loader that makes it true.

What lands

  • models/hf_fp8_blockwise.py — transformers' FineGrainedFP8 / DeepSeek-style layout: fp8 e4m3 [out,in] weights, a weight_scale_inv F32 grid at one scale per 128×128 block, applied as a multiplier (verified against transformers==5.13.1's own _dequantize_one, not inferred from the _inv suffix). resident=True keeps fp8 and runs the blockwise GEMM (fp8-w8a8-dynamic); resident=False upcasts ahead at load (fp8-w8a16, the portable/CPU arm).
  • @implements_contract(contract="hf.fp8-blockwise@1", …) on the loader, so the image's build-time derivation carries it into release_execution_lanes and the hub's Satisfies comparison can succeed for the first time.
  • inspect_hf_fp8_blockwise — verification from safetensors HEADERS ALONE, before any weight byte is read.
  • dequantize_block_scaled — the registry's declared hf.fp8_blockwise.dequant@1 reference dequant. §1.33 makes this the function a lossless conversion must prove bit-exactness against, so it exists once.

Why the refusals matter more than the load

cozy.fp8-rowwise@1 is also "fp8 e4m3 with dynamic activations" — same element type, same activation scheme, one English name. Its scale leaf is weight_scale, rank 1, per ROW. A decoder that reads a 128×128 reciprocal grid as a per-row multiplier broadcasts one scale across the wrong span: no exception, plausible numbers, wrong. So a rowwise tree is refused by name, and so are a transposed/mis-blocked grid, an fp8 weight with no scale, a rank-1 scale, and a config that disagrees with the bytes.

The pair is PRODUCIBLE, not CONVERTIBLE (DESIGN-RULINGS §1.33): re-blocking rowwise scales is a re-quantization with new numerics, so the remedy named in the refusal is a conversion-endpoint artifact, never a load-time repack.

No quantization happens in this path (Paul, 2026-08-11, th#1803 follow-up 2): quantization is ahead-of-time via conversion endpoints, never at cold boot. There is deliberately no quantize-if-missing fallback.

Proof (no mocks, CPU, ~9s)

tests/test_hf_fp8_blockwise_th1803.py builds a real tiny Llama, quantizes it to the layout, writes real safetensors + the same quantization_config shape the published conditioner carries, and loads it back through the declaring loader:

  1. the tree verifies from headers, with the block grid derived per unit (down_proj [128,256] → scale [1,2]);
  2. loaded weights are bit-identical to the reference dequant, and within fp8 error of the pre-quantization weights — which is what separates the multiply convention from its reciprocal (a divide is ~1e5 out, not 4%);
  3. a real forward produces finite logits;
  4. rowwise / transposed-grid / dense trees are each refused, by name;
  5. derive_execution_lanes() carries hf.fp8-blockwise@1 with this decoder.

Files: one new module, one new test, one line-group in tensor_layout_contract.py's handle list. No sibling lane touches these paths.

Follow-up in this lane: the decode proof on the real 27.6 GB artifact on a pod, recorded on th#1803.

…ES hf.fp8-blockwise@1

tensorhub 226070fd made the tensor-layout contract enforceable at rebind, and it
refused the H3 fp8 conditioner correctly: no decoder in any image declared
`hf.fp8-blockwise@1`, the contract transcribed from that artifact's own header
bytes. This is the declaration, and the loader that makes it true.

`models/hf_fp8_blockwise.py` reads transformers' FineGrainedFP8 layout — fp8
e4m3 [out,in] weights with a `weight_scale_inv` F32 grid, one scale per 128x128
block, applied as a MULTIPLIER (verified against transformers 5.13.1's own
`_dequantize_one`) — resident for the blockwise GEMM (fp8-w8a8-dynamic) or
upcast-ahead at load (fp8-w8a16).

The refusal is the point. `cozy.fp8-rowwise@1` is also "fp8 e4m3, dynamic
activations" with a per-ROW rank-1 `weight_scale`; reading one as the other
broadcasts a scale over the wrong span and yields plausible, wrong numbers with
no error. So the tree is verified from HEADERS ALONE before any weight byte is
read, and a rowwise tree is refused by name — PRODUCIBLE, not CONVERTIBLE
(DESIGN-RULINGS §1.33): re-blocking is a re-quantization, so the remedy is a
conversion-endpoint artifact, never a load-time repack.

`dequantize_block_scaled` is the registry's declared `hf.fp8_blockwise.dequant@1`
reference dequant — the function a §1.33 lossless conversion proves bit-exactness
against. No quantization happens here and there is no quantize-if-missing
fallback (Paul, 2026-08-11: quantization is ahead-of-time via conversion
endpoints, never at cold boot).

Proven with no mocks: a real tiny Llama quantized to the layout, written as
safetensors, loaded back through the declaring loader; weights bit-identical to
the reference dequant and within fp8 error of the originals, a real forward with
finite logits, rowwise/transposed/dense trees refused by name, and the build
derivation carrying the contract so `Satisfies` can succeed for the first time.
@PaulFidika
PaulFidika merged commit 153c51d into master Aug 11, 2026
2 checks passed
@PaulFidika
PaulFidika deleted the 1803-blockwise-loader branch August 11, 2026 21:03
PaulFidika added a commit that referenced this pull request Aug 11, 2026
th#1803 PR #662 landed `hf.fp8-blockwise@1` in the transcription WITH a real
decoder behind it while this branch was open. Rebased onto it; the changelog
no longer claims a transcription this issue did not make.
PaulFidika added a commit that referenced this pull request Aug 11, 2026
…de can execute (#663)

* pgw#1143 steps 1-3 (§1.33): a slot DECLARES the tensor layouts its code can execute

The code side of the layout contract had no declaration surface at all —
`api/slot.py` carried `pipeline_cls, selected_by, family, default_checkpoint,
root, optional` and nothing else, so the hub's th#1803 rebind gate was keyed
off the wheel's decoder census (2 distinct signatures across 64 releases)
rather than off what the author says the slot needs.

`Slot(layouts={component_path: (handle, ...)})` is that declaration: an
ordered tuple per component path, `"*"` for the whole tree, order IS
preference (§1.33 pt 2). It is published into the release manifest as
`functions[].slots[].layouts` for the hub to gate against (th#1809 T1-T2).

- **Absent is UNDECLARED**, a tri-state — never "accepts everything" and
  never "accepts nothing". An empty mapping and an empty tuple are both
  decoration-time errors; collapsing the tri-state is th#1580's fail-open
  defect wearing a new name.
- **On `Slot`, not `Compile`.** `Compile`'s fields feed `contract_axes()`,
  a cell-key input, and §1.33 pt 5 is that conversion is upstream of compute
  and invisible to cell identity. A layout declaration there would re-key
  every cell in the fleet for the one fact that must never enter the key.
- **Handles at the constructor, KEYS at decoration.** The handle vocabulary
  is `KNOWN_CONTRACTS` (refusal text reused verbatim from
  `tensor_layout_contract._validate`); the component keys are checked
  against the DERIVED component tree in `_spec_for_handler`, because a key
  that matches nothing reads as a declaration and gates nothing.
- **The SDK emits HANDLES, never digests** (th#1580 A2: descriptors are Go).
  The hub resolves handle -> `Contract.Digest()` at manifest ingest against
  its own registry — the only moment one wheel and one hub are both pinned —
  so `KNOWN_CONTRACTS` is honestly a transcription that may be stale and is
  CHECKED. `hf.fp8-blockwise@1` joins it: registered hub-side since th#1803
  (`226070fd`), and its absence made §1.33's own worked example (an fp8
  text_encoder against a bf16 DiT) undeclarable.
- **A composite `<topology>+<quant>` handle is refused, naming th#1809 T3.**
  The topology axis has no registry to compare against yet; storing half a
  pair is not exact.
- **The census stays a LOWER bound.** A declared handle no
  `@implements_contract` decoder backs lands on the manifest as
  `layouts_census_unbacked` and does NOT refuse — the blockwise conditioner
  is decoded natively by `transformers` through `quantization_config` with
  zero cozy markers, and refusing would make the motivating case illegal.
- **`scripts/lint_layout_declarations.py`** (new `fast gates` step) refuses a
  declaration the AST sweep cannot read. Discovery imports endpoint modules
  torch-stubbed, and pgw#1107's sweep reads them without importing at all; a
  computed declaration is legible to neither, which would make the published
  manifest the only place the demand can be read.

RED on master, measured: `Slot(P, layouts=...)` -> `TypeError: unexpected
keyword argument 'layouts'`; `_slot_to_manifest` emits
`['components', 'name', 'pipeline_class']`; `KNOWN_CONTRACTS` holds 5 of the
hub's 6 handles.

* pgw#1143: the layout lint sweeps src/ only

The fence's subject is a declaration that gets PUBLISHED. tests/ holds the
opposite by construction — a test proving the constructor refuses a computed
declaration has to write one — so sweeping it made this fence and its own
negative test mutually exclusive (CI, run 31534100939). Endpoint repos point
the script at their own tree, where every declaration is a real one.

* pgw#1143: pin the wire the hub actually reads

The declaration crosses TWO encoders before any gate sees it — msgspec TOML
into endpoint.lock, then JSON into tensorhub's `manifestSlotLayoutDoc`
(`map[string][]string`). A nested table inside an array-of-tables is exactly
where a TOML encoder's key ordering bites, and the failure would be silent:
the hub would read the slot as UNDECLARED and the gate would fall back to the
census with nothing to say about it.

* pgw#1143: drop the KNOWN_CONTRACTS claim — master got there first

th#1803 PR #662 landed `hf.fp8-blockwise@1` in the transcription WITH a real
decoder behind it while this branch was open. Rebased onto it; the changelog
no longer claims a transcription this issue did not make.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant