One photo of a room in, an editable 3D scene out. Runs on your own GPU and exports to the browser (WebXR), Blender, MuJoCo and anything that reads glTF.
Each photo next to the scene WorldBuilder built from it. Every object pops in on its own because it is its own mesh; the largest are switched off and on again. Shown after a few fixes in the browser viewer. Photos from Spacejoy and Jonathan Borba on Unsplash.
Give WorldBuilder a single ordinary photo of an indoor space. It finds the furniture and objects in it, builds a 3D model of each one, works out where the floor, walls and ceiling are, and puts everything back where it was in the photo. The result is a scene you can walk through in a browser or a VR headset, rearrange, or load into a simulator.
- Separate, named objects. Each chair, table or lamp is its own textured mesh, not one fused surface, so it can be moved, removed or given physics.
- A room around them. Floor, walls and ceiling are fitted to the photo and textured from it, with the hidden parts filled in.
- Fully local. The models run on your own NVIDIA GPU. A cloud detector (Gemini) is optional.
- Made to be used elsewhere. Blender add-on, MuJoCo export for robotics, a lightweight GLB for game engines, and a browser viewer for review and quick fixes.
Under the hood it chains open models: a vision-language model (Qwen3-VL or Gemini) lists the objects, SAM 3 segments every instance, SAM 3D Objects turns each mask into a mesh, and MoGe gives a point map of the whole scene. WorldBuilder's own code is what happens after that: room layout, placement with physical support, room textures and exports.
| Photo, the scene from the photo's viewpoint, and from the side |
|---|
![]() |
![]() |
More examples are coming. To make your own animation or stills from a scene:
python scripts/make_hero.py --scene outputs/<scene> photo.jpg --out hero.webp --stills <folder>.
| Guide | What's in it |
|---|---|
| Setup | Requirements, installing both environments, downloading the models, first run |
| Usage | The CLI, the web app, reviewing and editing scenes in the viewer, output files |
| Plugins | Blender, MuJoCo, Unreal, and where plugins are going next |
| Architecture | How the pipeline works stage by stage, file formats, coordinate frames, repo layout |
| Hardware | Measured GPU memory and time per stage, which cards work, what runs without a GPU |
| Benchmarks | How quality, speed and memory are measured and how two runs are compared |
| Troubleshooting | Common setup and runtime errors |
| Contributing | How to help, development setup, ground rules |
| Changelog | What changed in each release |
photo ─▶ 1 detect ─────▶ 2 segment ─────▶ 3 reconstruct ─────▶ 4 assemble ─────────────▶ viewer.html
Qwen3-VL or SAM 3: a mask SAM 3D Objects: layout, placement, scene_lite.glb
Gemini: what per instance, a mesh and pose room textures, exports Blender, MuJoCo
objects are plus floor, wall, per mask; one MoGe (CPU only, replayable)
in the photo ceiling, window point map per scene
- Detect. A vision-language model lists the kinds of objects worth reconstructing, with a short description and an expected count for each.
- Segment. SAM 3 turns each label into one mask per instance, and also segments the floor, walls, ceiling and windows.
- Reconstruct. SAM 3D Objects lifts each mask into a textured mesh with a pose. The scene point map is computed once and shared by every object.
- Assemble. WorldBuilder fits the floor, gravity and walls to the point map; places each object so its silhouette matches the photo and it rests on something (floor, a table, a wall); shares one good mesh between repeated objects such as a row of chairs; textures the room from the photo; and writes the viewer, the GLBs and optional MuJoCo files.
Stage 4 needs no GPU and can be replayed in seconds from what stage 3 leaves on disk, which is where most development happens. Details: Architecture.
You need Linux, an NVIDIA GPU with 24 GB or more (developed on an RTX 5090; 24 GB cards should work but haven't been tested), about 60 GB of disk, conda, and a Hugging Face account with access to the two gated SAM checkpoints. Request SAM 3 and SAM 3D Objects access first; approval can take a day.
The install takes one to two hours, mostly downloads. In outline:
git clone https://github.com/AdamRolander/WorldBuilder.git && cd WorldBuilder
git clone https://github.com/facebookresearch/sam3.git sam3_repo
git clone https://github.com/facebookresearch/sam-3d-objects.git sam3d_objects_repo
git clone https://github.com/facebookresearch/pytorch3d.git pytorch3d
# create the worldbuilder-main (and optionally worldbuilder-vlm) conda envs,
# then download the checkpoints: see docs/SETUP.md for the exact commandsThen:
conda activate worldbuilder-main
python main.py --image my_room.jpg --detector gemini # or --detector qwen, fully local
python -m http.server -d outputs/my_room_g 8000 # open http://localhost:8000/viewer.htmlA photo takes about seven minutes on an RTX 5090, most of it in SAM 3D (a few seconds per
object). Prefer a browser UI? python -m webapp.server serves an upload page at
http://localhost:5174. The full walkthrough is in Setup and
Usage.
No GPU? Placement, the room, the viewer and every exporter run on a CPU, and the test suite needs no models. See CPU-Only Development.
| Blender 4.2+ | Generate or import a scene from the sidebar; every object arrives as a named, textured mesh. Guide |
| MuJoCo / MJX | A metric, Z-up model with collision geometry and free joints on movable objects, plus a Gymnasium wrapper to put a robot in the room. Guide |
| Unreal Engine 5.4+ | Early scaffold, not yet tested. Guide |
| Everything else | scene_lite.glb (7-30 MB, baked textures) opens in Godot, Unity, three.js and most 3D tools |
Plugins are thin clients of a server you run; the models never leave it. Planned targets (Godot, Isaac Sim, Unity, USD, ROS 2 and more) are listed in Plugins.
WorldBuilder is alpha software. Things to know before relying on it:
- The back of every object is invented. SAM 3D completes what the photo didn't show; it is plausible, not measured. The same goes for walls the photo didn't see.
- Scale is an estimate. A single photo has no absolute scale. With the optional MoGe-2 step it is measured by a metric depth model; otherwise it assumes a typical camera height.
- Detection varies. The object list comes from a vision-language model and can miss things or differ between runs.
- Glass and mirrors are handled poorly; outdoor scenes are early.
- It needs a big GPU. 24 GB is the minimum; the models are run one at a time to fit.
Under active development. Releases and their changes are in the Changelog; plans and known problems are in the issue tracker. Bug reports with a photo are the most useful thing you can send.
Contributions are welcome, and much of the work (placement, textures, viewer, exporters,
plugins, docs) needs no GPU. Start with Contributing and issues labelled
good first issue or no-gpu.
If you use WorldBuilder in research, please cite it using the metadata in CITATION.cff (GitHub's Cite this repository button gives BibTeX and APA). A software paper is in preparation.
WorldBuilder's code is MIT licensed (see LICENSE), except the Blender add-on in
integrations/blender_worldbuilder/, which is GPL-3.0-or-later as Blender's extension
platform requires (see its licence). It runs on models and code that
you obtain separately under their own terms and that this project never redistributes:
SAM 3 and SAM 3D Objects (Meta, SAM License), MoGe (MIT), PyTorch3D (BSD), Qwen3-VL
(Apache-2.0). Thanks to their authors for releasing them.


