PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation

Haofei Xu^1,2,3 · Rundi Wu¹ · Philipp Henzler¹ · Nikolai Kalischek¹ · Michael Oechsle¹
Fabian Manhardt¹ · Marc Pollefeys^2,4 · Andreas Geiger^3,5 · Federico Tombari^1,6 · Michael Niemeyer¹

¹Google   ²ETH Zurich   ³University of Tübingen, Tübingen AI Center   ⁴Microsoft   ⁵KE:SAI   ⁶TUM

ICML 2026

Paper | Project Page | Models

PointDiT is a minimalist pixel-space Diffusion Transformer for monocular geometry estimation. It is a plain ViT that denoises raw 3D point map patches directly, conditioned on image tokens from a frozen pre-trained DINOv3: no point map tokenizer, no latent diffusion, no hybrid architecture, and no intricate loss formulation.

Installation

This codebase is developed with Python 3.12, PyTorch 2.7.0, and CUDA 12.8.

We recommend setting up a virtual environment (e.g., conda or venv) before installation:

# conda
conda create -y -n pointdit python=3.12
conda activate pointdit

# or venv
# python -m venv /path/to/venv/pointdit
# source /path/to/venv/pointdit/bin/activate

# torch 2.7.0, cuda 12.8
pip install torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128

pip install -r requirements.txt

# DINOv3 encoder code (not vendored; built with torch.hub from a local checkout)
git clone https://github.com/facebookresearch/dinov3.git third_party/dinov3

All commands are run from the repository root, and the launch scripts in scripts/.

The DINOv3 weights are gated and cannot be redistributed, so they are not part of the released checkpoints. Request access on the official DINOv3 repository and place the weights under pretrained/dinov3/ with their exact upstream filenames, or point DINOV3_WEIGHTS_DIR at the directory holding them. They are needed for evaluation and the demo as well as for training; see MODELS.md for the file list.

Model Zoo

Pre-trained models are available in the Model Zoo, at 256×256 and 512×512 for each of PointDiT-B / L / H. All weights are hosted on Hugging Face.

Download the weights and place (or symlink) them in the pretrained directory:

ln -s YOUR_MODEL_PATH pretrained

The checkpoints ship without the frozen DINOv3 encoder, which is downloaded separately (see Installation).

Demo

Check scripts/demo_l_512.sh for running our pre-trained models on your own images, with no ground truth and no camera intrinsics required. Set IMAGE_DIR at the top of the script and run it:

bash scripts/demo_l_512.sh

Any .jpg/.png files work, and nested folders are fine, since the loader globs recursively. Results are written to generation/.

Dataset Preparation

See DATASETS.md for detailed instructions on preparing the 12 training datasets and the 7 zero-shot evaluation datasets. We have provided the download link for the 7 evaluation datasets in DATASETS.md.

Symlink the prepared datasets to the datasets directory:

ln -s YOUR_DATASET_PATH datasets

Point clouds are never stored on disk: the training loader reads (RGB, depth, intrinsics) and back-projects on the fly.

Evaluation

Evaluation scripts are also provided in scripts/ for reproducing the results in our paper:

bash scripts/eval_l_512.sh

Swap l for b or h, and 512 for 256. Each script's header comment lists the numbers it should reproduce.

Training

PointDiT is trained in two stages: (1) 256×256 pre-training on SceneNet-RGBD and (2) 512×512 fine-tuning on the 11-dataset mixture.

bash scripts/train_stage1_256_b.sh
bash scripts/train_stage2_512_b.sh

The training scripts in scripts/ contain the exact commands and hyperparameters used for the experiments in our paper. Please refer to them for detailed configurations. Before training, you need to download the DINOv3 weights (see Installation) and prepare the datasets per DATASETS.md.

License

This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.

This codebase is released under the Apache License 2.0 (see the LICENSE file at the root of the repository). It also contains code derived from other open-source projects, which remains under its original licenses; see THIRD_PARTY_NOTICES for the full list and notices.

Citation

If you find this work useful, please consider citing:

@inproceedings{xu2026pointdit,
  title={PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation},
  author={Xu, Haofei and Wu, Rundi and Henzler, Philipp and Kalischek, Nikolai and Oechsle, Michael and Manhardt, Fabian and Pollefeys, Marc and Geiger, Andreas and Tombari, Federico and Niemeyer, Michael},
  booktitle={ICML},
  year={2026}
}

Acknowledgements

Our codebase builds upon several excellent open-source projects: JiT, DINOv3, MoGe, Depth Pro and utils3d. We thank all the authors for their great work.