Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The pipeline starts at the identity seed camera, then visits each supplied camer

## Environment and assets

Use the [unified GPU environment](/docs/maintainers/contributing). `gsplat` needs a complete CUDA development toolkit for its first extension build, including `nvcc` and `cuda_runtime.h`; set `CUDA_HOME` to that toolkit if auto-discovery selects an incomplete installation. Triton computes opacity-crossing median depth from the same gsplat projections and tile lists.
Use the [unified GPU environment](/docs/maintainers/contributing). `gsplat` needs a complete CUDA development toolkit for its first extension build, including `nvcc` and `cuda_runtime.h`; set `CUDA_HOME` to that toolkit if auto-discovery selects an incomplete installation. gsplat supplies differentiable projections and sorted tile lists. Triton composites RGB/alpha and computes opacity-crossing median depth with a shared alpha cap of 0.99. A surface contributes before its transmission triggers early termination.

WonderWorld uses shared released component weights rather than a separate trained WonderWorld generator checkpoint. The default checkpoint layout is:

Expand Down Expand Up @@ -82,7 +82,7 @@ worldfoundry-eval run wonderworld --print-config --json \

## Validation

Component contracts cover FP32/BF16 DDIM and unscaled-Euler trajectories, schedule boundaries, stateless guidance probes, real small U-Net channel ordering, VAE posterior/RNG ordering, depth decoding, pinned recipes, and shared codec ownership. Analytic geometry contracts cover perspective pixel covariance, tilted ground planes, off-axis pixels, and camera rotation. Reconstruction contracts cover equal-depth compositing, bounded trainable ranges, frozen-prefix/suffix preservation, pruning-state migration, nonmutation of input scenes, and median-depth traversal after exactly 50% transmission. Public contracts exercise standard evaluation invocation, image aliases, camera controls, generation-parameter precedence, lazy loading, input preflight, and failed/cancelled initialization cleanup.
Component contracts cover FP32/BF16 DDIM and unscaled-Euler trajectories, schedule boundaries, stateless guidance probes, real small U-Net channel ordering, VAE posterior/RNG ordering, depth decoding, pinned recipes, and shared codec ownership. Analytic geometry contracts cover perspective pixel covariance, tilted ground planes, off-axis pixels, and camera rotation. Reconstruction contracts cover equal-depth compositing, bounded trainable ranges, frozen-prefix/suffix preservation, pruning-state migration, nonmutation of input scenes, median-depth traversal after exactly 50% transmission, opaque-tail compositing, subpixel tile-boundary support, and RGB/alpha gradients. Continuation contracts cover fully covered views, current-camera conditioning, known-depth anchors, sky handling, and generation masks. Public contracts exercise standard evaluation invocation, image aliases, camera controls, generation-parameter precedence, lazy loading, input preflight, and failed/cancelled initialization cleanup.

The checkpoint-backed component contract checks every loaded tensor against its BF16-cast safetensors value, then compares native conditioning, initialization, U-Net prediction, scheduler update, and decoding against the corresponding network/schedule primitives. It also exercises decoded-depth gradients with frozen model weights:

Expand Down Expand Up @@ -116,8 +116,8 @@ The replay resolves the catalog loader and executes the standard evaluation invo
## Limitations

- Unrestricted CUDA decoded-depth gradients and photometric fitting do not guarantee bitwise repeatability. Small geometry differences can change disocclusion masks and subsequent generated regions. The exact A/B/A replay uses the documented MATH/deterministic-preference configuration and `refinement_steps=0`.
- The translated/rolled room replay has visible boundary holes and fragmented surfaces; visual quality is not accepted. Native artifact validation, released-stage numerical comparison, independent official full-pipeline parity and performance acceptance are separate gates.
- Native reconstruction uses gsplat RGB, opacity-crossing median depth, `0.8 L1 + 0.2 DSSIM`, and opacity/extent pruning. The pinned FLAGS fork regularizes covariance differently in forward and backward and can miss a later median-depth crossing after exactly 50% transmission. Native gradients are checked by finite differences and median depth by opacity-compositing fixtures, not fitted-output bitwise equivalence. Panorama generation, GPT prompts and the upstream browser server are outside this offline API. `stream()` is unsupported.
- The translated/rolled room replay has visible boundary holes and fragmented surfaces. Scene-quality observations are separate from implementation validation; native request/artifact contracts and released-stage numerical behavior are tested independently of visual preferences. Independent official full-pipeline parity and performance remain unvalidated.
- Native reconstruction uses gsplat projections, native RGB/alpha compositing, opacity-crossing median depth, `0.8 L1 + 0.2 DSSIM`, and opacity/extent pruning. The pinned FLAGS fork regularizes covariance differently in forward and backward and can miss a later median-depth crossing after exactly 50% transmission. Its subpixel tile bounds can also omit valid boundary contributions. Native gradients are checked by finite differences and median depth by opacity-compositing fixtures, not fitted-output bitwise equivalence. Panorama generation, GPT prompts and the upstream browser server are outside this offline API. `stream()` is unsupported.
- The pinned upstream repository publishes inference source but does not identify a root software license. Public availability is not an OSI-license claim. SD2 weights retain Open RAIL++-M terms; Marigold, OneFormer, RepViT-SAM, and gsplat retain their component terms. The catalog records component-specific licensing.

<ModelRelatedRecipes modelId="wonderworld" locale="en" />
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ WorldFoundry 已将 **WonderWorld** 接入仓库内的离线图像到三维世

## 环境与权重

使用[统一 GPU 环境](/docs/maintainers/contributing)。`gsplat` 首次构建扩展需要完整 CUDA 开发工具链,包括 `nvcc` 和 `cuda_runtime.h`;若自动发现的安装不完整,通过 `CUDA_HOME` 指向完整工具链。Triton 从同一份 gsplat 投影和瓦片列表计算透明度越过 50% 时的中位深度。
使用[统一 GPU 环境](/docs/maintainers/contributing)。`gsplat` 首次构建扩展需要完整 CUDA 开发工具链,包括 `nvcc` 和 `cuda_runtime.h`;若自动发现的安装不完整,通过 `CUDA_HOME` 指向完整工具链。gsplat 提供可微投影和排序后的瓦片列表,Triton 合成 RGB/透明度并计算透明度中位深度,共用 0.99 的透明度上限。触发透射率提前终止的表面仍参与本次合成。

WonderWorld 使用已经公开的共享组件权重,不需要单独训练的 WonderWorld 生成器检查点。默认目录布局:

Expand Down Expand Up @@ -82,7 +82,7 @@ worldfoundry-eval run wonderworld --print-config --json \

## 验证

组件契约覆盖 FP32/BF16 的 DDIM 与不缩放输入的 Euler 轨迹、排程边界、无状态引导探测、真实小 U-Net 的通道拼接、VAE 后验与随机数顺序、深度解码、固定配方和共享编解码器所有权。解析几何契约覆盖透视像素协方差、倾斜地面、离轴像素和相机旋转。重建契约覆盖同深度合成顺序、限定可训练区间、冻结前后缀保留、剪枝状态迁移、输入场景不变性,以及恰好 50% 透射率后的中位深度遍历。公共契约执行标准评估 invocation,检查图像别名、相机控制、生成参数优先级、延迟加载、输入预检及加载失败或取消时的清理。
组件契约覆盖 FP32/BF16 的 DDIM 与不缩放输入的 Euler 轨迹、排程边界、无状态引导探测、真实小 U-Net 的通道拼接、VAE 后验与随机数顺序、深度解码、固定配方和共享编解码器所有权。解析几何契约覆盖透视像素协方差、倾斜地面、离轴像素和相机旋转。重建契约覆盖同深度合成顺序、限定可训练区间、冻结前后缀保留、剪枝状态迁移、输入场景不变性,以及恰好 50% 透射率后的中位深度遍历、高不透明度尾部合成、亚像素瓦片边界覆盖和 RGB/透明度梯度。续建契约覆盖完全可见视角、当前相机的条件图、已有深度锚点、天空处理和生成掩码。公共契约执行标准评估 invocation,检查图像别名、相机控制、生成参数优先级、延迟加载、输入预检及加载失败或取消时的清理。

真实权重组件契约逐张量核对加载值与 safetensors 转换为 BF16 后的值,并将原生条件编码、初始化、U-Net 预测、调度更新和解码与对应网络及调度原语比较,同时执行冻结模型权重下的解码深度梯度检查:

Expand Down Expand Up @@ -116,8 +116,8 @@ python tests/manual/wonderworld_inference.py \
## 限制

- 不约束后端时,CUDA 解码深度梯度和光度拟合不保证逐位重复;几何的细微差异可能改变显露区域掩码,进而改变后续生成内容。精确 A/B/A 回放使用上述 MATH/确定性优先配置及 `refinement_steps=0`。
- 平移/滚转房间回放仍有可见的边界孔洞和表面碎裂,视觉质量未验收。原生产物验证、发布阶段数值对照、独立官方端到端对齐与性能验收属于不同门禁。
- 原生重建使用 gsplat RGB、透明度中位深度、`0.8 L1 + 0.2 DSSIM` 及透明度/尺度剪枝。固定版本 FLAGS 的协方差正向与反向正则项不一致,且可能漏掉恰好 50% 透射率之后的中位深度跨越。原生梯度通过有限差分校验,中位深度通过透明度合成样例验证,不以拟合产物逐位一致作为证明。全景生成、GPT 提示词和上游浏览器服务器不属于此离线 API。暂不支持 `stream()`。
- 平移/滚转房间回放存在可见的边界孔洞和表面碎裂。场景质量观察与实现验证分开记录;原生请求/产物契约及发布阶段数值行为独立于视觉偏好进行测试。独立官方端到端对齐与性能尚未验证。
- 原生重建使用 gsplat 投影、原生 RGB/透明度合成、透明度中位深度、`0.8 L1 + 0.2 DSSIM` 及透明度/尺度剪枝。固定版本 FLAGS 的协方差正向与反向正则项不一致,且可能漏掉恰好 50% 透射率之后的中位深度跨越;其亚像素瓦片包围盒也可能漏掉边界上的有效贡献。原生梯度通过有限差分校验,中位深度通过透明度合成样例验证,不以拟合产物逐位一致作为证明。全景生成、GPT 提示词和上游浏览器服务器不属于此离线 API。暂不支持 `stream()`。
- 固定版本的上游仓库公开了推理源码,但没有标明根目录软件许可证。公开可获取不等于采用 OSI 许可证。SD2 权重保留 Open RAIL++-M 条款;Marigold、OneFormer、RepViT-SAM、gsplat 保留各自条款,目录按组件记录许可。

<ModelRelatedRecipes modelId="wonderworld" locale="zh" />
15 changes: 8 additions & 7 deletions docs/fumadocs/lib/model-recipes-data.json
Original file line number Diff line number Diff line change
Expand Up @@ -111484,13 +111484,13 @@
"Camera paths use row-major OpenCV world-to-camera matrices; named upstream viewer matrices use column-major 16-vectors.",
"Exported scene depth equals Marigold scene depth times 1000; camera translations use these exported units.",
"Guided depth uses the final eight Euler steps and known scene/ground anchors; latent noise and U-Net input remain unscaled as in the pinned source.",
"gsplat RGB rendering, opacity-crossing median depth, and 0.8 L1 plus 0.2 DSSIM fitting are native implementations; full official rasterizer and quality parity remain pending.",
"gsplat supplies differentiable projections and sorted tile lists; native Triton RGB/alpha and median depth share alpha cap 0.99 and retain the surface crossing the termination threshold.",
"Fitting uses 0.8 L1 plus 0.2 DSSIM. Integration contracts and numerical comparison are separate from scene-quality observations; full official pipeline parity remains pending.",
"New Gaussians precede the frozen scene at equal depth; only their bounded particle range is fitted and pruned.",
"Upstream reports 48 GB VRAM. Local component weights and a CUDA-compatible gsplat installation are required.",
"No separate WonderWorld generator checkpoint is needed. SD2 uses CreativeML Open RAIL++-M; retain all component licensing terms.",
"Uses the unified WorldFoundry GPU environment and an ABI-compatible gsplat build.",
"All inference orchestration is in-tree; no external checkout, PyTorch3D, Flask server, or OpenAI credential is needed.",
"Stage the public SD2 Inpainting, Marigold depth/normals, OneFormer ADE20K metadata/weights, and RepViT-SAM weights outside the source tree."
"All inference orchestration is in-tree; no external checkout, PyTorch3D, Flask server, or OpenAI credential is needed."
]
},
"sources": [
Expand Down Expand Up @@ -111709,15 +111709,16 @@
}
],
"notes": [
"Native SD2 inpainting and Marigold recipes, canonical OneFormer/RepViT-SAM perception, guided depth, occlusion layers, and gsplat Gaussian reconstruction.",
"Native SD2 inpainting and Marigold recipes, canonical OneFormer/RepViT-SAM perception, guided depth, occlusion layers, and Gaussian reconstruction with gsplat projections and native compositing.",
"Finite offline image/camera-path requests export standard .splat artifacts without an external checkout or upstream server.",
"Full official FLAGS rasterizer/panorama parity and interactive browser acceptance remain separate validation scopes.",
"Native standard-runner recipes own SD2/Marigold diffusion; no upstream checkout, Diffusers pipeline, Flask server, or PyTorch3D extension is required.",
"The single RGB image is resized to 512x512. Each optional camera matrix requests one new world view, starting from an identity seed camera.",
"Camera paths use row-major OpenCV world-to-camera matrices; named upstream viewer matrices use column-major 16-vectors.",
"Exported scene depth equals Marigold scene depth times 1000; camera translations use these exported units.",
"Guided depth uses the final eight Euler steps and known scene/ground anchors; latent noise and U-Net input remain unscaled as in the pinned source.",
"gsplat RGB rendering, opacity-crossing median depth, and 0.8 L1 plus 0.2 DSSIM fitting are native implementations; full official rasterizer and quality parity remain pending.",
"gsplat supplies differentiable projections and sorted tile lists; native Triton RGB/alpha and median depth share alpha cap 0.99 and retain the surface crossing the termination threshold.",
"Fitting uses 0.8 L1 plus 0.2 DSSIM. Integration contracts and numerical comparison are separate from scene-quality observations; full official pipeline parity remains pending.",
"New Gaussians precede the frozen scene at equal depth; only their bounded particle range is fitted and pruned.",
"Upstream reports 48 GB VRAM. Local component weights and a CUDA-compatible gsplat installation are required.",
"No separate WonderWorld generator checkpoint is needed. SD2 uses CreativeML Open RAIL++-M; retain all component licensing terms."
Expand All @@ -111736,10 +111737,10 @@
"WonderWorld 从单张图像创建分层高斯世界,并在指定相机视角扩展场景。WorldFoundry 使用原生扩散配方与树内场景重建,导出标准 .splat 场景和视角预览。"
],
"architecture": [
"Guided Marigold depth diffusion aligns generated views to existing geometry and a ground plane. RepViT-SAM refines region disparity, OneFormer identifies sky and occluders, SD2 completes hidden/new regions, and normal-oriented Gaussians are reconstructed and fitted with gsplat."
"Guided Marigold depth diffusion aligns generated views to existing geometry and a ground plane. RepViT-SAM refines region disparity, OneFormer identifies sky and occluders, SD2 completes hidden/new regions, and normal-oriented Gaussians use gsplat projections and native differentiable RGB/alpha compositing for reconstruction and fitting."
],
"architectureZh": [
"引导式 Marigold 深度扩散将新视角对齐已有几何与地面。RepViT-SAM 细化区域视差,OneFormer 识别天空与遮挡物,SD2 补全隐藏和新出现区域,再通过 gsplat 重建并拟合法线定向的高斯。"
"引导式 Marigold 深度扩散将新视角对齐已有几何与地面。RepViT-SAM 细化区域视差,OneFormer 识别天空与遮挡物,SD2 补全隐藏和新出现区域,再通过 gsplat 投影与原生可微 RGB/透明度合成重建并拟合法线定向的高斯。"
],
"usageNotes": [
"Supply an RGB image and local SD2 Inpainting, Marigold depth/normals, OneFormer, and RepViT-SAM assets. Optional camera_path contains OpenCV world-to-camera matrices in exported scene units. Default image size is 512x512; calls own independent scenes while retaining weights."
Expand Down
Loading
Loading