diff --git a/CHANGELOG.md b/CHANGELOG.md index d76a94017..c19c36a6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,10 +10,19 @@ adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `gmlx pull`, `image generate`, `serve` and the ComfyUI pack run FLUX.3 klein and FLUX.3 klein Base GGUFs, with edits from up to 10 reference images. +- `gmlx pull`, `image generate`, `serve` and the ComfyUI pack run FLUX.2 klein + 4B and FLUX.2 klein Base 4B GGUFs, with edits from up to 10 reference images. +- `gmlx pull`, `image generate`, `serve` and the ComfyUI pack run FLUX.2 klein + 9B and FLUX.2 klein Base 9B GGUFs, licensed for non-commercial use only. +- `gmlx pull`, `image generate`, `serve` and the ComfyUI pack run FLUX.2 klein + 9B KV GGUFs, which edit from two 1024x1024 references in about half the + 9B's time. - `context_copy: true` or `GMLX_CONTEXT_COPY=1` drafts from the prompt when the output repeats it, which speeds up edits and quotes. - Muse Spark (GGUF arch `muse-spark`), with reasoning effort intents, tool calls and image and audio input through its `--mmproj` companion. +- An image LoRA whose PEFT header sets `use_rslora` loads, scaled as PEFT + scales it, where it was refused. ### Changed @@ -25,9 +34,15 @@ adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). runs faster on NAX GPUs (M5). - With `--stream-experts`, a prompt that uses most of a layer's experts prefills faster. `GMLX_ARENA_SPLIT_MAX_SHARE` tunes the choice. +- `gmlx pull` of an image model reuses a VAE GGUF on disk packed for its + family from another source, and names it, where it downloaded the VAE again. ### Fixed +- A streamed image request refused while it ran, such as one too large for + memory, got a server error frame where the same request unstreamed got a 400. +- The `gmlx pull` plan of an image model could say its parts fit in stages + when `image generate` then refused the image, as it counted file bytes alone. - A DFlash drafter that took speculation back up after a batch shrank within the width cap drafted from stale context, so almost no draft was accepted. - A sampled request whose structured output allowed fewer tokens than its @@ -52,6 +67,10 @@ adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). `serve`, and `response_format` constrains only the final answer. - `gmlx launch` could miss a SIGTERM or SIGHUP on a busy Mac, so a stop or a closed terminal did not end the session until the client exited. +- `gmlx Sigmas` in ComfyUI gave Qwen-Image 2.1 ComfyUI's simple schedule. It + takes the image's width and height and gives the schedule of that size. +- An image request that failed on an internal error got a 400 and left no + trace in the log. It gets a 500, and the server logs its trace. ## [0.5.0] - 2026-10-12 diff --git a/README.md b/README.md index 247e11203..f42f37437 100644 --- a/README.md +++ b/README.md @@ -98,9 +98,9 @@ The recording runs at true speed, with a 27B model in a local server. - Embeddings, rerank, speech-to-text and text-to-speech on the same server, for a local RAG and voice stack. See [Speech, embeddings and rerank](https://asher.github.io/gmlx/services.html). -- Image generation from Z-Image, Qwen-Image 2.1, Krea 2 Turbo and FLUX.3 - klein GGUFs, on the command line, on the OpenAI image routes and in - ComfyUI. It needs an M3 chip or newer. See +- Image generation from Z-Image, Qwen-Image 2.1, Krea 2 Turbo, FLUX.2 klein + and FLUX.3 klein GGUFs, on the command line, on the OpenAI image routes + and in ComfyUI. It needs an M3 chip or newer. See [Image generation](https://asher.github.io/gmlx/images.html). - A decision API at `/v1/systemone` that returns a probability for each answer to yes-or-no, choice and score questions about a text. See diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 539ad3ff6..43ebb5e5b 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -22,6 +22,7 @@ in [`licenses/`](licenses/). | [mflux](https://github.com/filipstrand/mflux) | MIT | The Z-Image transformer at commit `e066d98`, ported to `gmlx/image/families/z_image/model.py` with the fused qkv kept and modules named after the GGUF tensors, and its attention, timestep and RoPE helpers in `gmlx/image/ops/attention.py` and `gmlx/image/ops/embeds.py`. | | [diffusers](https://github.com/huggingface/diffusers) | Apache-2.0 | Code at commit `e0abab8`, ported to MLX and modified for gmlx: the Z-Image forward of `transformer_z_image.py` in `gmlx/image/families/z_image/model.py`, its timestep embedding, RoPE tables and rotary product in `gmlx/image/ops/embeds.py` and `gmlx/image/ops/attention.py`, the RMSNorm rounding of `normalization.py` in `gmlx/image/ops/norms.py`, and the FLUX.1 autoencoder of `autoencoders/vae.py`, `resnet.py`, `unet_2d_blocks.py`, `downsampling.py`, `upsampling.py` and `attention_processor.py` in `gmlx/image/vae/flux1_ae.py`, the quant convs and BatchNorm statistics of `autoencoders/autoencoder_kl_flux2.py` in `gmlx/image/vae/flux2_ae.py`, the Qwen-Image 2.1 autoencoder of `autoencoders/autoencoder_kl_qwenimage21.py`, for one frame, in `gmlx/image/vae/qwen_image21_vae.py`, and the UniPC bh2 sampler of `schedulers/scheduling_unipc_multistep.py` in `gmlx/image/unipc.py`. Also the Qwen-Image 2.1 forward, token layout and RoPE of `transformer_qwenimage21.py` in `gmlx/image/families/qwen_image21/model.py` and `layout.py`, and the dynamic shift of `pipelines/qwenimage21/pipeline_qwenimage21.py` and `schedulers/scheduling_flow_match_euler_discrete.py` in `gmlx/image/schedule.py`, and the reference image sizes, the prompt template with image slots and the image branch of the encoder tap of that pipeline in `gmlx/image/families/qwen_image21/edit.py`. For Krea 2, at the same commit: the forward of `transformer_krea2.py` in `gmlx/image/families/krea2/model.py`, the Qwen-Image autoencoder of `autoencoders/autoencoder_kl_qwenimage.py`, for one frame, in `gmlx/image/vae/qwen_image_vae.py`, the prompt template, encoder layers and constants of `pipelines/krea2/pipeline_krea2.py` and `modular_pipelines/krea2/` in `gmlx/image/families/krea2/family.py`, and the schedule of that pipeline in `krea2_sigmas` of `gmlx/image/schedule.py`. | | [flux-3-image-klein](https://github.com/black-forest-labs/flux-3-image-klein) | Apache-2.0 | BFL's FLUX.3 klein reference at commit `45e57c9`, ported to MLX and modified for gmlx: the DiT constants and tensor layout of `src/flux3/dit.py`, the image RoPE scale and schedule alpha of `src/flux3/pipeline.py` and the encoder layers, pad multiple, length, pad token and position and chat template of `src/flux3/text_encoder.py` in `gmlx/image/families/flux3_klein/family.py`, the DiT forward of `src/flux3/dit.py` with the position ids of `pipeline.py`, `autoencoder.py` and `text_encoder.py` in `gmlx/image/families/flux3_klein/model.py`, the schedule of `src/flux3/sampling.py` in `flux3_sigmas` of `gmlx/image/schedule.py`, the latent patchify and BatchNorm of `src/flux3/autoencoder.py` in `gmlx/image/vae/flux2_ae.py`, and the reference canvas, reference preparation and default size of `src/flux3/pipeline.py` with the reference choice and output crop of `src/flux3/generate.py` in `gmlx/image/families/flux3_klein/edit.py`. | +| [flux2](https://github.com/black-forest-labs/flux2) | Apache-2.0 | BFL's FLUX.2 reference at commit `50fe516`, ported to MLX and modified for gmlx: the DiT constants of `Klein4BParams`, `Klein9BParams` and `Flux2Params` and the tensor names and shapes of `Flux2` in `src/flux2/model.py`, the sampling defaults of `FLUX2_MODEL_INFO` in `src/flux2/util.py`, the prompt rules of `Qwen3Embedder` in `src/flux2/text_encoder.py`, the rounding of `denoise` and `denoise_cfg` and the reference image rules of `encode_image_refs` in `src/flux2/sampling.py` in `gmlx/image/families/flux2_klein/family.py`, the constants of `Klein9BParams`, the 9B rows of `FLUX2_MODEL_INFO` and the Qwen3-8B encoder of `load_qwen3_embedder` in `src/flux2/text_encoder.py` in `gmlx/image/families/flux2_klein_9b/family.py`, the `flux.2-klein-9b-kv` row of `FLUX2_MODEL_INFO` in `gmlx/image/families/flux2_klein_9b_kv/family.py` and the step of `denoise_cached` in `src/flux2/sampling.py` in `gmlx/image/families/flux2_klein_9b_kv/step.py`, the DiT forward of `src/flux2/model.py`, with `forward_kv_extract`, `forward_kv_cached` and `causal_attn_fn`, and the token ids of `prc_img`, `prc_txt` and `encode_image_refs` in `src/flux2/sampling.py` in `gmlx/image/families/flux2_klein/model.py`, and the schedule of `compute_empirical_mu` and `get_schedule` in `src/flux2/sampling.py` in `flux2_mu` and `flux2_sigmas` of `gmlx/image/schedule.py`, and the reference image preparation of `encode_image_refs` and `default_prep` in `src/flux2/sampling.py` in `gmlx/image/families/flux2_klein/edit.py`. | | [ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF) | Apache-2.0 | The model detection keys of `convert.py` at commit `6ea2651`, which `gmlx/image/fingerprint.py` uses to refuse other diffusion model layouts by name. | | [misaki](https://github.com/hexgrad/misaki) | Apache-2.0 | Whole package vendored at `gmlx/_vendor/misaki` (commit `fba1236`): Kokoro's grapheme-to-phoneme front-end, used by the TTS service until upstream's next PyPI release. | | [musl](https://musl.libc.org) | MIT | Statically linked into the guest entry binary `gmlx/container/guest/gmlx-entry` as its C library ([`licenses/musl-COPYRIGHT`](licenses/musl-COPYRIGHT)). | diff --git a/docs/cli.md b/docs/cli.md index 57ca0cad6..a91a5922b 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -639,13 +639,27 @@ gmlx pull qwen-image --quant Q5_K_M ``` An image model name, `z-image-turbo`, `qwen-image`, `qwen-image-turbo`, -`krea-2-turbo`, `flux-3-klein` or `flux-3-klein-base`, pulls its DiT, text -encoder and VAE, checks the VAE -against its published sha256, packs it to GGUF, and adds an `image_models` -entry. The quants default to -the ones for this Mac's memory, and an encoder or VAE already on disk is -used again when its weights match. -[Models and quants](images.md#models-and-quants) has the quants for each Mac. +`krea-2-turbo`, `flux-2-klein`, `flux-2-klein-base`, `flux-2-klein-9b`, +`flux-2-klein-base-9b`, `flux-2-klein-9b-kv`, `flux-3-klein` or +`flux-3-klein-base`, pulls its DiT, text encoder and VAE, checks the VAE +against its published sha256, packs it to GGUF, and adds an +`image_models` entry. The quants default to +the ones for this Mac's memory, and an encoder already on disk is used +again when its weights match. On a Mac where no DiT of the model makes an +image, such as a 9B model on 16 GB, `pull` downloads nothing, names the +smallest Mac that runs one, and takes a DiT only when `--quant` names it. +When the three parts do not fit the Mac's memory together, or an image +does not fit at all, the plan gives the peak of an image at the entry's +size, as `image generate` plans it. A VAE GGUF on disk packed from the recipe's +source is used again. Without one, a pack from another source serves when +it is labeled for the family or for one that decodes with the same +autoencoder, such as an earlier F32 pack of the FLUX.2 VAE, and the plan +names the pack it reuses. +[Models and quants](images.md#models-and-quants) has the quants for each +Mac. `pull` prints each model's license before it downloads, and the +license of `flux-2-klein-9b`, `flux-2-klein-base-9b` and +`flux-2-klein-9b-kv` allows non-commercial use only, as +[Licenses](images.md#licenses) says. Of several encoders on disk, `pull` takes the quant for this Mac first, then a chat model, then the file nearest the DiT. A chat model taken as @@ -663,7 +677,7 @@ names the encoder file and the vision tower that `pull` adds. | `--hf-source ID` | None | Treat the architecture as loadable with this config override. | | `--max-mb N` | `128` | Cap the header range read. | | `--json` | Off | Emit each verdict as JSON before downloading. | -| `--quant QUANT` | The one for this Mac's memory | Image models: download the DiT at this quant. | +| `--quant QUANT` | The one for this Mac's memory | Image models: download the DiT at this quant. Without it, a pull on a Mac where no DiT of the model makes an image downloads nothing. | | `--encoder-quant QUANT` | The one for this Mac's memory | Image models: download the text encoder at this quant. Without it, a matching encoder on disk is used at any quant. | | `--only PARTS` | All three | Image models: pull only these parts, a comma list of `dit`, `encoder` and `vae`. | | `--keep-source` | Off | Image models: keep the safetensors VAE after packing it. | @@ -958,11 +972,18 @@ the weights load. | Qwen-Image 2.1 | Qwen3-VL-8B, its text GGUF | The Qwen-Image 2.1 autoencoder | 40 steps and guidance 1. Turbo: 8 steps and guidance 1. | | Krea 2 | Qwen3-VL-4B-Instruct, its text GGUF | The Qwen-Image autoencoder | Turbo: 8 steps and guidance 1. | | FLUX.3 klein | Qwen3-VL-4B-Instruct, its text GGUF | The FLUX.2 autoencoder | 16 steps and guidance 1. Base: 50 steps and guidance 3.5. | +| FLUX.2 klein | Qwen3-4B | The FLUX.2 autoencoder | 4 steps and guidance 1, which is the only guidance it takes. Base: 50 steps and guidance 4. | +| FLUX.2 klein 9B | Qwen3-8B | The FLUX.2 autoencoder | The same as FLUX.2 klein's. Its license allows non-commercial use only. | FLUX.3 klein Base shares the distilled model's architecture, encoder and VAE, and gmlx tells the two apart by their tensors. The distilled model runs any step count on its own schedule. +FLUX.2 klein Base and the distilled model are told apart the same way, +at both sizes. The distilled model refuses `--guidance` other than 1 before any file +loads, as BFL's command line does, and names Base as the checkpoint that +takes guidance. + Qwen-Image 2.1 Turbo shares the base model's architecture, encoder and VAE. gmlx tells the two apart by their weights. Turbo samples on the eight sigmas it was distilled to, and refuses any other step count, on the command @@ -985,9 +1006,10 @@ multiple of 32, and a reference keeps its alpha channel. The text encoder reads the references through its vision tower, which `--encoder-mmproj` loads from the encoder's mmproj GGUF, and the VAE encodes them. `--size` sets the size of the output, which defaults to the entry's `defaults.size` -or the variant's, and not to a reference's. The fit check counts the references, and a request that does -not fit is refused before any model loads. Z-Image and Krea 2 take no -reference images, and `--ref` exits 2 on them. +or the variant's, and not to a reference's. The fit check counts the +references, and a request that does not fit is refused before any model +loads. Z-Image and Krea 2 take no reference images, and `--ref` exits 2 on +them. `--ref` gives FLUX.3 klein up to 10 reference images too. Its text encoder reads the prompt alone, so it needs no `--encoder-mmproj`, and the VAE @@ -995,17 +1017,33 @@ encodes the references for the DiT. A single reference is scaled down to at most the area of 2048x2048 pixels and padded out to sides that are multiples of 16. Of several, each is scaled down to at most the area of 1024x1024 and cropped to sides that are multiples of 16. Without `--size` -the output has the first reference's size, and the output of a single -reference is cropped back to the scaled image. FLUX.3 klein sizes its -references by this rule, and `--ref-size` exits 2 on it. +the output has the first reference's size as prepared, scaled into 4096 +pixels a side and 2048x2048 pixels at its shape when it is past them, and +the output of a single reference is cropped back to the scaled image. +FLUX.3 klein sizes its references by this rule, and `--ref-size` exits 2 +on it. + +`--ref` gives FLUX.2 klein up to 10 reference images as well, read as +FLUX.3 klein reads them, with no `--encoder-mmproj`. A single reference is +scaled down to at most the area of 2024x2024 pixels, and each of several +to at most the area of 1024x1024. Each is then cropped to sides that are +multiples of 16, around the center. A reference needs sides of at least 64 +pixels, and neither side more than 8 times the other. Without `--size` the +output has the first reference's size as prepared, scaled into the same +limits, and it is not cropped back, so an 8000x1000 reference, prepared at +5712x704, gives a 4096x496 output. `--ref-size` exits 2 on FLUX.2 klein +too. `--adapter` applies a LoRA safetensors file to the DiT, live beside the quantized weights, so the file is read once and nothing is merged. The -kohya, diffusers, PEFT and OneTrainer key spellings are read. The strength -follows the path after a colon and defaults to 1, and it may be negative. -Repeat the flag to stack adapters, whose deltas add. A LoHa, LoKr, DoRA, -full-weight or Tucker LoCon file is refused by name from its header, before -any model loads. So is a file for another model family or one with keys +kohya, diffusers, PEFT and OneTrainer key spellings are read. The rank and +alpha in a diffusers file's PEFT header scale each pair as the diffusers +loader scales it, rank-stabilized LoRA included. ComfyUI does not read that +header, so such a file runs at another strength there, and gmlx warns of +it. The strength follows the path after a colon and defaults to 1, and it +may be negative. Repeat the flag to stack adapters, whose deltas add. A +LoHa, LoKr, DoRA, full-weight or Tucker LoCon file is refused by name from +its header, before any model loads. So is a file for another model family or one with keys that match no module of the DiT, which its GGUF header names. An adapter's text encoder pairs are applied to the text encoder at the same strength. @@ -1030,7 +1068,7 @@ and `crop`, the left, top, right and bottom of the image in it. | `--prompt TEXT` | Required, or `--prompt-file` | Generate from this prompt. | | `--prompt-file FILE` | None | Read the prompt from this file. | | `--negative-prompt TEXT` | Empty | Guide away from this prompt, or from the empty prompt without it. It is unused when `--guidance` is 1. | -| `--size WxH` | The init image's, else a FLUX.3 klein reference's, else the entry's `defaults.size`, else the variant's | Set the width and height, each a multiple of 32 on Qwen-Image 2.1 and of 16 on the others. Another size exits 2 and names the multiple. | +| `--size WxH` | The init image's, else a FLUX.2 klein or FLUX.3 klein reference's, else the entry's `defaults.size`, else the variant's | Set the width and height, each a multiple of 32 on Qwen-Image 2.1 and of 16 on the others. Another size exits 2 and names the multiple. | | `--steps N` | The variant's | Sample for N steps. | | `--seed N` | Random, and printed | Seed the first image. | | `--guidance CFG` | The variant's | Set classifier-free guidance. 1 turns it off. | @@ -1038,7 +1076,7 @@ and `crop`, the left, top, right and bottom of the image in it. | `--init-image PATH` | None | Start from this image, resized to `--size` (img2img). | | `--mask PATH` | None | Change only where this mask is white (inpainting). It needs `--init-image`. | | `--strength F` | `0.6`, or `1` with `--mask` | Set how far img2img moves from the init image, from 0.01 to 1. | -| `--ref PATH` | None | Edit from this reference image, on Qwen-Image 2.1 and FLUX.3 klein. Repeat the flag for up to 10 images, in the order the prompt names them. | +| `--ref PATH` | None | Edit from this reference image, on Qwen-Image 2.1, FLUX.2 klein and FLUX.3 klein. Repeat the flag for up to 10 images, in the order the prompt names them. | | `--ref-size N` | `1024` | On Qwen-Image 2.1, resize each reference to about N x N pixels at its own aspect ratio, from 256 to 1024. It needs `--ref`. | | `--count N` | `1` | Generate N images. | | `--output PATH` | `./gmlx-