Skip to content
Merged
15 changes: 8 additions & 7 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: diffuseR
Title: Functional Interface to Diffusion Models in R
Version: 0.2.0
Version: 0.2.1.1
Authors@R: c(
person("Troy", "Hernandez", email = "troy@cornball.ai", role = c("aut", "cre"),
comment = c(ORCID = "0009-0005-4248-604X")),
Expand All @@ -9,12 +9,13 @@ Authors@R: c(
comment = "portions ported from the diffusers library (Apache-2.0); see inst/COPYRIGHTS"),
person("Lightricks Ltd.", role = "cph",
comment = "LTX checkpoint layout and pipeline constants; see inst/COPYRIGHTS"))
Description: A native R implementation of diffusion models providing a functional
interface to state-of-the-art generative AI. Inspired by Hugging Face's Python
'diffusers' library, 'diffuseR' allows users to generate and manipulate images
using text prompts through models like Stable Diffusion without Python dependencies.
The package provides a streamlined, idiomatic R experience with support for multiple
diffusion schedulers and device acceleration.
Description: A native R implementation of diffusion models providing a
functional interface to state-of-the-art generative AI. Inspired by the
'Python' library 'diffusers' from 'Hugging Face'
<https://huggingface.co/>, 'diffuseR' generates and manipulates images
from text prompts using models such as 'Stable Diffusion', with no
'Python' dependency. Supports multiple diffusion schedulers and device
acceleration.
License: Apache License (>= 2)
URL: https://github.com/cornball-ai/diffuseR
BugReports: https://github.com/cornball-ai/diffuseR/issues
Expand Down
43 changes: 43 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,46 @@
# diffuseR 0.2.1.1

Addresses the CRAN review of the 0.2.0 submission.

* Every exported `.Rd` with a `\usage` block now documents its return
value: 50 `@return` tags added, chiefly to the `nn_module` generators
for the FLUX, FLUX.2, Z-Image, LTX-2.3 and Gemma3 ports.
* Examples: 14 of the 23 `\dontrun{}` blocks now run during check, and
were rewritten to be self-contained instead of referencing undefined
objects. The 9 that remain need model weights on disk and are
itemised in `cran-comments.md`.
* `ddim_scheduler_create()` was uncallable at its documented defaults:
`beta_schedule` was never passed through `match.arg()`, so `switch()`
errored on the length-3 default, and the `device` default was a
length-2 vector that `torch_tensor()` rejects. `ddim_scheduler_step()`
had the same missing `match.arg()` on `prediction_type`. Every
internal caller passed these explicitly, so the broken defaults went
unnoticed. `device` now defaults to `torch_device("cpu")`.
* `DESCRIPTION`: software names single-quoted ('Python', 'Stable
Diffusion', 'Hugging Face' with its URL) and the trailing whitespace
that had been folding into double spaces since the first commit
removed.
* `save_video()`'s mp4 example is no longer live: the encoder inherits
the session's stdin, which `R CMD check --as-cran` uses to feed the
example script to R, so it consumed part of the script.

# diffuseR 0.2.0.1

* The FLUX-family image loaders (`flux_load_pipeline`,
`flux2_load_pipeline`, `zimage_load_pipeline`) now page-lock the
phase-swapped transformer, VAE decoder, and text encoder(s) at load,
so the per-generation CPU<->GPU moves run at DMA rate (offload becomes
a pointer swap). A new `pin` argument, `NULL` by default, resolves via
`options(diffuseR.pin_staging)` then the host-RAM-aware `recommend()`
decision. Resident-fp8 transformers (flux2/zimage) stage their fp8
weight fields too.
* `flux_load_pipeline()` GPU-encodes T5-XXL (bfloat16) on 14 GB+ cards,
where its encode phase fits; smaller cards keep the float32 CPU
encode. `text_device` defaults to `NULL` (resolved from the VRAM
tier). An explicit `text_device = "cpu"` still encodes in place.
* Internal: the pinned-staging helpers lost their `ltx23` prefix
(`staging.R`); `recommend()` and the loaders share one `.pin_decision`.

# diffuseR 0.2.0

## Serving
Expand Down
25 changes: 25 additions & 0 deletions R/audio_vae_ltx23.R
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ NULL
#' @param stride Integer.
#' @param causality_axis "height", "width", "width-compatibility", or "none".
#'
#' @return Module whose forward(x) returns the convolved tensor, padded
#' so that each output frame depends only on current and earlier
#' input frames.
#'
#' @export
ltx23_audio_causal_conv2d <- torch::nn_module(
"ltx23_audio_causal_conv2d",
Expand Down Expand Up @@ -63,6 +67,9 @@ ltx23_audio_causal_conv2d <- torch::nn_module(
#' @param in_channels,out_channels Integers.
#' @param causality_axis Character.
#'
#' @return Module whose forward(x) returns \code{x} plus the residual
#' branch, a tensor of the same shape as \code{x}.
#'
#' @export
ltx23_audio_resnet_block <- torch::nn_module(
"ltx23_audio_resnet_block",
Expand Down Expand Up @@ -109,6 +116,9 @@ ltx23_audio_resnet_block <- torch::nn_module(
#' @param in_channels Integer.
#' @param causality_axis Character.
#'
#' @return Module whose forward(x) returns the tensor upsampled 2x by
#' nearest-neighbour interpolation and convolved.
#'
#' @export
ltx23_audio_upsample <- torch::nn_module(
"ltx23_audio_upsample",
Expand Down Expand Up @@ -139,6 +149,9 @@ ltx23_audio_upsample <- torch::nn_module(
#' @param in_channels Integer.
#' @param causality_axis Character.
#'
#' @return Module whose forward(x) returns the strided convolution of
#' \code{x}, halving the downsampled axes.
#'
#' @export
ltx23_audio_downsample <- torch::nn_module(
"ltx23_audio_downsample",
Expand Down Expand Up @@ -168,6 +181,10 @@ ltx23_audio_downsample <- torch::nn_module(
#' See \code{\link{ltx23_audio_decoder}}.
#' @param in_channels Integer. Mel channels (2 = stereo).
#'
#' @return Module whose forward(x) returns the encoded audio latent, a
#' tensor downsampled along time and mel axes with the configured
#' latent channel count.
#'
#' @export
ltx23_audio_encoder <- torch::nn_module(
"ltx23_audio_encoder",
Expand Down Expand Up @@ -269,6 +286,9 @@ ltx23_audio_encoder <- torch::nn_module(
#' @param causality_axis Character.
#' @param mel_bins Integer. Output mel bins (crop/pad target).
#'
#' @return Module whose forward(x) returns the decoded mel
#' spectrogram reconstructed from an audio latent.
#'
#' @export
ltx23_audio_decoder <- torch::nn_module(
"ltx23_audio_decoder",
Expand Down Expand Up @@ -391,6 +411,11 @@ ltx23_audio_decoder <- torch::nn_module(
#' See \code{\link{ltx23_audio_decoder}}.
#' @param in_channels Integer. Mel input channels (2 = stereo).
#'
#' @return Module bundling the audio encoder and decoder. Its
#' forward(z) is \code{decode(z)}, returning the mel spectrogram for a
#' latent; \code{$encode()} and \code{$decode()} are callable
#' separately.
#'
#' @export
ltx23_audio_vae <- torch::nn_module(
"ltx23_audio_vae",
Expand Down
13 changes: 5 additions & 8 deletions R/auto_devices.R
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,13 @@
#' @export
#'
#' @examples
#' \dontrun{
#' # Auto-detect best configuration
#' devices <- auto_devices("sdxl")
#' # Force a strategy: no GPU or nvidia-smi needed.
#' str(auto_devices("sdxl", strategy = "cpu_only"))
#'
#' # Use with models2devices
#' m2d <- models2devices("sdxl", devices = auto_devices("sdxl"))
#' str(auto_devices("sd21", strategy = "unet_gpu"))
#'
#' # Force CPU-only
#' devices <- auto_devices("sdxl", strategy = "cpu_only")
#' }
#' # Auto-detect free VRAM and pick a strategy for this machine.
#' str(auto_devices("sdxl"))
auto_devices <- function(model = "sdxl", strategy = "auto") {
# Free-VRAM requirements in GB (float16 component sizes + overhead)
requirements <- list(sd21 = list(full_gpu = 4, unet_gpu = 3),
Expand Down
15 changes: 15 additions & 0 deletions R/connectors_ltx23.R
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ ltx23_per_token_rms_norm <- function(x, eps = 1e-6) {
#' @param rope_type "split" (LTX-2.3) or "interleaved".
#' @param num_attention_heads Integer. For the split per-head layout.
#'
#' @return Module whose forward(batch_size, pos, device) returns
#' \code{list(cos_freqs, sin_freqs)}, the 1-D rotary tables for a
#' sequence of length \code{pos}.
#'
#' @export
ltx23_rotary_pos_embed_1d <- torch::nn_module(
"ltx23_rotary_pos_embed_1d",
Expand Down Expand Up @@ -142,6 +146,11 @@ ltx23_transformer_block_1d <- torch::nn_module(
#' @param eps Numeric. Norm epsilon.
#' @param gated_attention Logical. Per-head attention output gates.
#'
#' @return Module whose forward(hidden_states, attention_mask,
#' attn_mask_binarize_threshold) returns
#' \code{list(hidden_states, attention_mask)}: the transformed
#' sequence and the (possibly binarized) mask that accompanies it.
#'
#' @export
ltx23_connector_transformer_1d <- torch::nn_module(
"ltx23_connector_transformer_1d",
Expand Down Expand Up @@ -266,6 +275,12 @@ ltx23_connector_transformer_1d <- torch::nn_module(
#' (DiT inner dims: 4096 / 2048).
#' @param proj_bias Logical. Projection bias (TRUE for LTX-2.3).
#'
#' @return Module whose forward(text_encoder_hidden_states,
#' attention_mask) returns \code{list(video_text_embedding,
#' audio_text_embedding, attention_mask)}: the caption states adapted
#' for the video and audio cross-attention streams, plus the binary
#' mask to use with them.
#'
#' @export
ltx23_text_connectors <- torch::nn_module(
"ltx23_text_connectors",
Expand Down
58 changes: 33 additions & 25 deletions R/ddim_scheduler.R
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,14 @@
#' \url{https://arxiv.org/abs/2010.02502}
#'
#' @examples
#' \dontrun{
#' # Create a DDIM scheduler with custom parameters
#' scheduler <- ddim_scheduler_create(
#' num_train_timesteps = 1000,
#' num_inference_steps = 30,
#' eta = 0.5,
#' beta_schedule = "scaled_linear"
#' )
#' if (torch::torch_is_installed()) {
#' scheduler <- ddim_scheduler_create(
#' num_train_timesteps = 1000,
#' num_inference_steps = 5,
#' eta = 0.5,
#' beta_schedule = "scaled_linear"
#' )
#' scheduler$timesteps
#' }
#' @export
ddim_scheduler_create <- function(num_train_timesteps = 1000,
Expand All @@ -60,7 +60,8 @@ ddim_scheduler_create <- function(num_train_timesteps = 1000,
beta_start = 0.00085, beta_end = 0.012,
rescale_betas_zero_snr = FALSE,
dtype = torch::torch_float32(),
device = c(torch::torch_device("cpu"), torch::torch_device("cuda"))) {
device = torch::torch_device("cpu")) {
beta_schedule <- match.arg(beta_schedule)
betas <- switch(beta_schedule,
"linear" = seq(beta_start, beta_end, length.out = num_train_timesteps),
"scaled_linear" = seq(sqrt(beta_start), sqrt(beta_end),
Expand Down Expand Up @@ -156,14 +157,18 @@ ddim_scheduler_create <- function(num_train_timesteps = 1000,
#' \url{https://arxiv.org/abs/2202.00512}
#'
#' @examples
#' \dontrun{
#' # Perform a denoising step
#' result <- ddim_scheduler_step(
#' model_output = model_output,
#' timestep = timestep,
#' sample = sample,
#' eta = 0, # Deterministic sampling
#' prediction_type = "epsilon")
#' if (torch::torch_is_installed()) {
#' scheduler <- ddim_scheduler_create(num_inference_steps = 5)
#' sample <- torch::torch_randn(c(1, 4, 8, 8))
#' model_output <- torch::torch_randn(c(1, 4, 8, 8))
#' result <- ddim_scheduler_step(
#' model_output = model_output,
#' timestep = scheduler$timesteps[1],
#' sample = sample,
#' schedule = scheduler,
#' eta = 0, # Deterministic sampling
#' prediction_type = "epsilon")
#' result$shape
#' }
#' @export
ddim_scheduler_step <- function(model_output, timestep, sample, schedule,
Expand All @@ -174,6 +179,7 @@ ddim_scheduler_step <- function(model_output, timestep, sample, schedule,
prediction_type = c("epsilon", "sample", "v_prediction"),
dtype = torch::torch_float32(),
device = "cpu") {
prediction_type <- match.arg(prediction_type)
# 1. get previous step value (= timestep + 1); i.e. python-indexing
timestep_index <- torch::torch_tensor(timestep + 1,
dtype = torch::torch_long(), device = torch::torch_device(device))
Expand Down Expand Up @@ -312,14 +318,16 @@ ddim_scheduler_step <- function(model_output, timestep, sample, schedule,
#' specified timestep, with beta being the noise schedule.
#'
#' @examples
#' \dontrun{
#' # Assuming we have latents, noise, and a scheduler
#' noised_latents <- scheduler_add_noise(
#' original_latents = latents,
#' noise = torch::torch_randn_like(latents),
#' timestep = scheduler$timesteps[1],
#' scheduler_obj = scheduler
#' )
#' if (torch::torch_is_installed()) {
#' scheduler <- ddim_scheduler_create(num_inference_steps = 5)
#' latents <- torch::torch_randn(c(1, 4, 8, 8))
#' noised_latents <- scheduler_add_noise(
#' original_latents = latents,
#' noise = torch::torch_randn_like(latents),
#' timestep = scheduler$timesteps[1],
#' scheduler_obj = scheduler
#' )
#' noised_latents$shape
#' }
#'
#' @export
Expand Down
12 changes: 12 additions & 0 deletions R/dit_flux2_modules.R
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ NULL
#' @param mod_param_sets Integer. Number of (shift, scale, gate) triples.
#' @param bias Logical.
#'
#' @return Module whose forward(temb) returns the modulation tensor
#' \code{linear(silu(temb))}, holding \code{mod_param_sets} triples of
#' (shift, scale, gate) along the last axis.
#'
#' @export
flux2_modulation <- torch::nn_module(
"flux2_modulation",
Expand Down Expand Up @@ -62,6 +66,9 @@ flux2_modulation <- torch::nn_module(
#' @param mult Numeric. Inner dim multiplier (FLUX.2: 3.0).
#' @param bias Logical.
#'
#' @return Module whose forward(x) returns the SwiGLU-gated projection
#' of \code{x}, a tensor with the last axis of width \code{dim_out}.
#'
#' @export
flux2_feed_forward <- torch::nn_module(
"flux2_feed_forward",
Expand Down Expand Up @@ -94,6 +101,11 @@ flux2_feed_forward <- torch::nn_module(
#' @param eps Numeric. RMS norm epsilon.
#' @param bias Logical.
#'
#' @return Module whose forward(hidden_states, image_rotary_emb,
#' chunk_size) returns the block output [B, S, query_dim]: attention
#' and MLP branches computed in parallel from one fused projection,
#' concatenated, and projected back by a second fused layer.
#'
#' @export
flux2_parallel_self_attention <- torch::nn_module(
"flux2_parallel_self_attention",
Expand Down
10 changes: 10 additions & 0 deletions R/dit_flux_modules.R
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ flux_ada_layer_norm_zero_single <- torch::nn_module(
#' @param bias Logical. Bias on the projection (TRUE for FLUX.1, FALSE
#' for FLUX.2).
#'
#' @return Module whose forward(x, cond) returns \code{x} normalized and
#' then scaled and shifted by the conditioning embedding, a tensor of
#' the same shape as \code{x}.
#'
#' @export
flux_ada_layer_norm_continuous <- torch::nn_module(
"flux_ada_layer_norm_continuous",
Expand Down Expand Up @@ -112,6 +116,12 @@ flux_ada_layer_norm_continuous <- torch::nn_module(
#' @param bias Logical. Bias on the linear projections (TRUE for FLUX.1,
#' FALSE for FLUX.2).
#'
#' @return Module whose forward(hidden_states, encoder_hidden_states,
#' image_rotary_emb, chunk_size) returns the attended image stream
#' [B, S, query_dim]. When \code{encoder_hidden_states} is supplied
#' (double-stream blocks) it returns \code{list(image, text)} instead,
#' each projected by its own output layer.
#'
#' @export
flux_attention <- torch::nn_module(
"flux_attention",
Expand Down
5 changes: 5 additions & 0 deletions R/dit_ltx23.R
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ NULL
#' @param gated_attn,cross_attn_mod,audio_gated_attn,audio_cross_attn_mod,perturbed_attn
#' LTX-2.3 feature flags (all TRUE for the 2.3 checkpoints).
#'
#' @return Module whose forward(hidden_states, ...) returns
#' \code{list(sample, audio_sample)}: the predicted velocity for the
#' video latent tokens and, when the audio branch is active, for the
#' audio latent tokens (\code{audio_sample} is NULL otherwise).
#'
#' @export
ltx23_transformer <- torch::nn_module(
"ltx23_transformer",
Expand Down
Loading
Loading