Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,12 @@ Bindings: `HF(id, revision=, dtype=, subfolder=, files=, storage_dtype=)`,
The slot name comes from the `models={}` key or the `setup()` parameter —
never a constructor argument. `storage_dtype="fp8"` keeps denoiser weights in
fp8-E4M3 storage with per-layer upcast to the compute `dtype` (half the VRAM
on any card); fp8-stored `#fp8` flavors get the same treatment automatically.
on any card); fp8-stored `#fp8` artifacts get the same treatment automatically.
Quantization itself is ahead-of-time only — a conversion endpoint produces the
artifact, never `setup()` (th#1803) — and `flavor=` is a dying axis: selection
within a tag group becomes tensor-layout-contract compatibility
(DESIGN-RULINGS §1.33, pgw#1143). See
[docs/endpoint-authoring.md](docs/endpoint-authoring.md).

Curated checkpoint selection is a runtime payload argument: a handler declares
`model: SomeModelChoice` (a `ModelChoice` enum of `Model` rows, each carrying a
Expand Down
5 changes: 4 additions & 1 deletion docs/compile-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,10 @@ about the WEIGHTS, not about the compiled program:
handles it implements with `@implements_contract`
(`gen_worker.models.tensor_layout_contract`), and the vocabulary itself lives
in tensorhub. It says what the bytes ARE and nothing about compilation.
(th#1580 / th#1721; was called "the artifact contract".)
(th#1580 / th#1721; was called "the artifact contract".) **th#1803 makes this
a core platform feature:** code is quant-generic over a declared layout, and
the platform answers compatibility ahead of time — at rebind and at request
time — never by loading and running on a rented pod.
- **The tensor-binding contract** — the artifact's LINKING rule for tensors:
bound by name at load (DYNAMIC — an opaque slot the compiler must never
value-specialize, which is what makes a cell checkpoint-agnostic) versus a
Expand Down
11 changes: 11 additions & 0 deletions docs/convert.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

Cozy Creator's model ETL: hub ingest (HF + Civitai), dtype cast / quantization, repackage, and Tensorhub publish.

> **This is where quantization happens — the only place (th#1803).** Paul, 2026-08-11: no
> inference-time quantization. Serving loads a pre-quantized artifact produced here; quantizing
> inside an endpoint's `setup()` is deleted, not kept as a fallback (it lengthens every cold boot
> and wastes transfer — fetch 30 GB of bf16, discard 15 GB).
>
> **`flavor` as a selector is dead** (DESIGN-RULINGS §1.33): what a producer emits is an artifact
> carrying a tensor-layout contract, and consumers select within a tag group by contract
> compatibility, not by an arbitrary `#flavor` string. The `ProducedFlavor` / `publish_flavors` /
> `#fp8` spellings below are the current API, not the target one — th#1809 (hub) and pgw#1143 (SDK)
> own the replacement.

- **Ingest**: HuggingFace (`HfApi.list_repo_files` + classifier + `snapshot_download(allow_patterns=…)`) and Civitai (bounded provider API).
- **Convert**: streaming dtype cast + fp8-E4M3 storage cast (`#fp8` flavor), bitsandbytes nf4/fp4, GGUF (llama.cpp toolchain), singlefile↔diffusers repackage.
- **Publish**: one commit call against Tensorhub's HF-shaped `/commits` write API. `mode` defaults to `replace` (th#1400 — a checkpoint is complete in itself); pass `mode="merge"` explicitly, and only when adding to an existing snapshot.
Expand Down
51 changes: 48 additions & 3 deletions docs/endpoint-authoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,11 @@ key-naming convention, file topology), named by a descriptor handle
`<producer>.<format>@<major>`. Layout says what the bytes ARE; binding says how
they are ADDRESSED at load.

**Declare the layout you execute, not a quant** (th#1803). Your code should be
quant-generic over its declared layout, so the platform can answer "will this
checkpoint work with this code?" before renting a pod — and so the owner can
rebind fp8↔bf16 as config. See "Do NOT quantize weights in `setup()`" below.

**You configure the compiler by how you write the code.** The classification is
not a setting supplied out of band — it derives from `state_dict` membership at
trace time. The code IS the configuration, and the assignment style below is the
Expand Down Expand Up @@ -257,8 +262,21 @@ the denoiser VRAM on any card, no fp8 silicon required. Snapshots whose
weights are already fp8-stored (an `#fp8` flavor) get the same treatment
automatically; endpoint code stays precision-agnostic and
`ModelEvent.vram_bytes` reports the measured resident size. Quantized
formats are platform-produced stored flavors (`#fp8`, `#nvfp4` on Blackwell)
— there is no runtime "quantize my model" kwarg. The one exception is the
formats are platform-produced stored artifacts (`#fp8`, `#nvfp4` on Blackwell)
— there is no runtime "quantize my model" kwarg, and which one a component
serves is deploy CONFIG, not a literal you pick here (th#980, th#1803).

> **`flavor` is being deleted (th#1803, DESIGN-RULINGS §1.33).** Paul: the
> flavor system was *"an arbitrary-string sub-selector within a tag-group…
> too imprecise."* Selection within a tag group becomes tensor-layout-contract
> compatibility — the endpoint declares a per-slot SET of accepted layouts and
> the platform grades each candidate COMPATIBLE / CONVERTIBLE / PRODUCIBLE /
> INCOMPATIBLE ahead of time. `#flavor` refs, flavored tag rows and the flavor
> columns go away with no alias. The replacement surfaces are being designed in
> th#1809 (hub) and pgw#1143 (SDK); the `flavor=`/`#fp8` spellings in this
> document describe what exists today, not what to build against.

The one exception is the
EMERGENCY rung (automatic on CUDA hosts): when
even the downloaded flavor cannot fit free VRAM, the loading layer
runtime-quantizes the denoiser to 4-bit nf4 with a loud warning (quality
Expand Down Expand Up @@ -687,7 +705,34 @@ not servable through these calls, and a green integrity result is **not** a
quality signal — a melted or over-smoothed render scores HIGHER on this
statistic than a clean one. See `gen_worker.output_integrity`.

## If `setup()` quantizes, report the lane it applied (pgw#1104)
## Do NOT quantize weights in `setup()` (th#1803)

**Component quant selection is BINDING CONFIG, not endpoint code.** fp8 vs
bf16 for a text encoder or a denoiser is a config record change — point the
component ref at the fp8 artifact — with no code change, no rebuild and no
redeploy, and it is overridable per request where the endpoint allows it.
Write `setup()` **quant-generically**: declare the tensor layout you execute
and serve whatever satisfies it. A `serve_recipe` that casts bf16 weights on
every cold boot, switchable only by shipping a new endpoint version, is the
rejected pattern (DESIGN-RULINGS §1.32).

**No inference-time quantization at all.** Quantization happens ahead of time —
a conversion endpoint produces the artifact — for two measured reasons: it
lengthens every cold boot, and it wastes transfer (download 30 GB of bf16 and
immediately discard 15 GB). So the recipe LOADS the bound pre-quantized
artifact; the boot-quant path is deleted, not kept as a fallback. (The fit
ladder's emergency nf4 rung below is a different thing — a last-resort OOM
degradation, not a selection mechanism.)

What stays in code: kernel selection, compile scope, allocator settings, the
warmup obligation. What leaves: the choice of which weights to run.

### If a recipe still converts weights, report the lane it applied (pgw#1104)

Two shipped endpoints predate the ruling. Until they are converted, the
reporting call below is mandatory for them — and it stays in the SDK
regardless, because runtime-gated *engagement* (a kernel or compile arm that
may or may not apply on this card) still has to be reported honestly.

A serve-time recipe — torchao `quantize_()`, an fp8 cast, anything that
converts the weights inside `setup()` — moves the lane the endpoint EXECUTES
Expand Down
Loading