From ea1fe39388c83e74a354f2988e079aee05f66066 Mon Sep 17 00:00:00 2001 From: Paul Fidika Date: Tue, 11 Aug 2026 13:33:11 -0600 Subject: [PATCH 1/3] th#1803 docs: bindings-as-config + tensor-layout contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The authoring guide told authors how to quantize in setup(); Paul rejected that pattern (DESIGN-RULINGS §1.32). Quant selection is binding config, code is quant-generic over its declared tensor layout. report_applied_lane stays — it now covers the runtime-gated engagement half, and the two un-converted endpoints. --- docs/compile-cache.md | 5 ++++- docs/endpoint-authoring.md | 29 +++++++++++++++++++++++++++-- 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/docs/compile-cache.md b/docs/compile-cache.md index b80c63d65..13abc459a 100644 --- a/docs/compile-cache.md +++ b/docs/compile-cache.md @@ -41,7 +41,10 @@ about the WEIGHTS, not about the compiled program: handles it implements with `@implements_contract` (`gen_worker.models.tensor_layout_contract`), and the vocabulary itself lives in tensorhub. It says what the bytes ARE and nothing about compilation. - (th#1580 / th#1721; was called "the artifact contract".) + (th#1580 / th#1721; was called "the artifact contract".) **th#1803 makes this + a core platform feature:** code is quant-generic over a declared layout, and + the platform answers compatibility ahead of time — at rebind and at request + time — never by loading and running on a rented pod. - **The tensor-binding contract** — the artifact's LINKING rule for tensors: bound by name at load (DYNAMIC — an opaque slot the compiler must never value-specialize, which is what makes a cell checkpoint-agnostic) versus a diff --git a/docs/endpoint-authoring.md b/docs/endpoint-authoring.md index d7a42b5b1..3a0bb7305 100644 --- a/docs/endpoint-authoring.md +++ b/docs/endpoint-authoring.md @@ -96,6 +96,11 @@ key-naming convention, file topology), named by a descriptor handle `.@`. Layout says what the bytes ARE; binding says how they are ADDRESSED at load. +**Declare the layout you execute, not a quant** (th#1803). Your code should be +quant-generic over its declared layout, so the platform can answer "will this +checkpoint work with this code?" before renting a pod — and so the owner can +rebind fp8↔bf16 as config. See "Do NOT quantize weights in `setup()`" below. + **You configure the compiler by how you write the code.** The classification is not a setting supplied out of band — it derives from `state_dict` membership at trace time. The code IS the configuration, and the assignment style below is the @@ -258,7 +263,8 @@ weights are already fp8-stored (an `#fp8` flavor) get the same treatment automatically; endpoint code stays precision-agnostic and `ModelEvent.vram_bytes` reports the measured resident size. Quantized formats are platform-produced stored flavors (`#fp8`, `#nvfp4` on Blackwell) -— there is no runtime "quantize my model" kwarg. The one exception is the +— there is no runtime "quantize my model" kwarg, and which flavor a component +serves is deploy CONFIG, not a literal you pick here (th#980, th#1803). The one exception is the EMERGENCY rung (automatic on CUDA hosts): when even the downloaded flavor cannot fit free VRAM, the loading layer runtime-quantizes the denoiser to 4-bit nf4 with a loud warning (quality @@ -687,7 +693,26 @@ not servable through these calls, and a green integrity result is **not** a quality signal — a melted or over-smoothed render scores HIGHER on this statistic than a clean one. See `gen_worker.output_integrity`. -## If `setup()` quantizes, report the lane it applied (pgw#1104) +## Do NOT quantize weights in `setup()` (th#1803) + +**Component quant selection is BINDING CONFIG, not endpoint code.** fp8 vs +bf16 for a text encoder or a denoiser is a config record change — point the +component ref at the fp8 artifact — with no code change, no rebuild and no +redeploy, and it is overridable per request where the endpoint allows it. +Write `setup()` **quant-generically**: declare the tensor layout you execute +and serve whatever satisfies it. A `serve_recipe` that casts bf16 weights on +every cold boot, switchable only by shipping a new endpoint version, is the +rejected pattern (DESIGN-RULINGS §1.32). + +What stays in code: kernel selection, compile scope, allocator settings, the +warmup obligation. What leaves: the choice of which weights to run. + +### If a recipe still converts weights, report the lane it applied (pgw#1104) + +Two shipped endpoints predate the ruling. Until they are converted, the +reporting call below is mandatory for them — and it stays in the SDK +regardless, because runtime-gated *engagement* (a kernel or compile arm that +may or may not apply on this card) still has to be reported honestly. A serve-time recipe — torchao `quantize_()`, an fp8 cast, anything that converts the weights inside `setup()` — moves the lane the endpoint EXECUTES From 9719f7fb0717864fe304cf045dfd100f032659a5 Mon Sep 17 00:00:00 2001 From: Paul Fidika Date: Tue, 11 Aug 2026 14:25:57 -0600 Subject: [PATCH 2/3] th#1803 follow-ups: no inference-time quantization, and flavor-as-selector is dead MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Paul's second and third same-day follow-ups. Quantization is ahead-of-time only (conversion endpoint) — boot-quant costs cold-boot seconds and wasted transfer, and the path is deleted rather than kept as a fallback. The flavor system is an arbitrary-string sub-selector being replaced by tensor-layout-contract compatibility (DESIGN-RULINGS §1.33; th#1809 hub, pgw#1143 SDK), so the flavor spellings in these docs are marked as current-state, not the target. --- docs/convert.md | 11 +++++++++++ docs/endpoint-authoring.md | 26 +++++++++++++++++++++++--- 2 files changed, 34 insertions(+), 3 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index 3e6b8084d..497bf19a5 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -2,6 +2,17 @@ Cozy Creator's model ETL: hub ingest (HF + Civitai), dtype cast / quantization, repackage, and Tensorhub publish. +> **This is where quantization happens — the only place (th#1803).** Paul, 2026-08-11: no +> inference-time quantization. Serving loads a pre-quantized artifact produced here; quantizing +> inside an endpoint's `setup()` is deleted, not kept as a fallback (it lengthens every cold boot +> and wastes transfer — fetch 30 GB of bf16, discard 15 GB). +> +> **`flavor` as a selector is dead** (DESIGN-RULINGS §1.33): what a producer emits is an artifact +> carrying a tensor-layout contract, and consumers select within a tag group by contract +> compatibility, not by an arbitrary `#flavor` string. The `ProducedFlavor` / `publish_flavors` / +> `#fp8` spellings below are the current API, not the target one — th#1809 (hub) and pgw#1143 (SDK) +> own the replacement. + - **Ingest**: HuggingFace (`HfApi.list_repo_files` + classifier + `snapshot_download(allow_patterns=…)`) and Civitai (bounded provider API). - **Convert**: streaming dtype cast + fp8-E4M3 storage cast (`#fp8` flavor), bitsandbytes nf4/fp4, GGUF (llama.cpp toolchain), singlefile↔diffusers repackage. - **Publish**: one commit call against Tensorhub's HF-shaped `/commits` write API. `mode` defaults to `replace` (th#1400 — a checkpoint is complete in itself); pass `mode="merge"` explicitly, and only when adding to an existing snapshot. diff --git a/docs/endpoint-authoring.md b/docs/endpoint-authoring.md index 3a0bb7305..f348d8752 100644 --- a/docs/endpoint-authoring.md +++ b/docs/endpoint-authoring.md @@ -262,9 +262,21 @@ the denoiser VRAM on any card, no fp8 silicon required. Snapshots whose weights are already fp8-stored (an `#fp8` flavor) get the same treatment automatically; endpoint code stays precision-agnostic and `ModelEvent.vram_bytes` reports the measured resident size. Quantized -formats are platform-produced stored flavors (`#fp8`, `#nvfp4` on Blackwell) -— there is no runtime "quantize my model" kwarg, and which flavor a component -serves is deploy CONFIG, not a literal you pick here (th#980, th#1803). The one exception is the +formats are platform-produced stored artifacts (`#fp8`, `#nvfp4` on Blackwell) +— there is no runtime "quantize my model" kwarg, and which one a component +serves is deploy CONFIG, not a literal you pick here (th#980, th#1803). + +> **`flavor` is being deleted (th#1803, DESIGN-RULINGS §1.33).** Paul: the +> flavor system was *"an arbitrary-string sub-selector within a tag-group… +> too imprecise."* Selection within a tag group becomes tensor-layout-contract +> compatibility — the endpoint declares a per-slot SET of accepted layouts and +> the platform grades each candidate COMPATIBLE / CONVERTIBLE / PRODUCIBLE / +> INCOMPATIBLE ahead of time. `#flavor` refs, flavored tag rows and the flavor +> columns go away with no alias. The replacement surfaces are being designed in +> th#1809 (hub) and pgw#1143 (SDK); the `flavor=`/`#fp8` spellings in this +> document describe what exists today, not what to build against. + +The one exception is the EMERGENCY rung (automatic on CUDA hosts): when even the downloaded flavor cannot fit free VRAM, the loading layer runtime-quantizes the denoiser to 4-bit nf4 with a loud warning (quality @@ -704,6 +716,14 @@ and serve whatever satisfies it. A `serve_recipe` that casts bf16 weights on every cold boot, switchable only by shipping a new endpoint version, is the rejected pattern (DESIGN-RULINGS §1.32). +**No inference-time quantization at all.** Quantization happens ahead of time — +a conversion endpoint produces the artifact — for two measured reasons: it +lengthens every cold boot, and it wastes transfer (download 30 GB of bf16 and +immediately discard 15 GB). So the recipe LOADS the bound pre-quantized +artifact; the boot-quant path is deleted, not kept as a fallback. (The fit +ladder's emergency nf4 rung below is a different thing — a last-resort OOM +degradation, not a selection mechanism.) + What stays in code: kernel selection, compile scope, allocator settings, the warmup obligation. What leaves: the choice of which weights to run. From e35061a6eae929b3c8081e7e48df1f54615692ac Mon Sep 17 00:00:00 2001 From: Paul Fidika Date: Tue, 11 Aug 2026 14:43:53 -0600 Subject: [PATCH 3/3] =?UTF-8?q?th#1803:=20README=20binding=20blurb=20?= =?UTF-8?q?=E2=80=94=20ahead-of-time=20quantization=20only,=20flavor=3D=20?= =?UTF-8?q?is=20a=20dying=20axis?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 8ab0802cd..b6597d67d 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,12 @@ Bindings: `HF(id, revision=, dtype=, subfolder=, files=, storage_dtype=)`, The slot name comes from the `models={}` key or the `setup()` parameter — never a constructor argument. `storage_dtype="fp8"` keeps denoiser weights in fp8-E4M3 storage with per-layer upcast to the compute `dtype` (half the VRAM -on any card); fp8-stored `#fp8` flavors get the same treatment automatically. +on any card); fp8-stored `#fp8` artifacts get the same treatment automatically. +Quantization itself is ahead-of-time only — a conversion endpoint produces the +artifact, never `setup()` (th#1803) — and `flavor=` is a dying axis: selection +within a tag group becomes tensor-layout-contract compatibility +(DESIGN-RULINGS §1.33, pgw#1143). See +[docs/endpoint-authoring.md](docs/endpoint-authoring.md). Curated checkpoint selection is a runtime payload argument: a handler declares `model: SomeModelChoice` (a `ModelChoice` enum of `Model` rows, each carrying a