CDIS: Cross-Dimensional Class-Agnostic 3D Instance Segmentation via 2D Mask Tracking and 3D-2D Projection Merging
Juno Kim1* · Hye-Jung Yoon1* · Yesol Park1* · Byoung-Tak Zhang1
1Interdisciplinary Program in AI, Seoul National University
*Equal contribution
IEEE/RSJ IROS 2025
TL;DR: CDIS produces globally consistent, class-agnostic 3D instance segmentation by explicitly tracking 2D instance masks across frames and associating them with 3D superpoints — a feedback loop between 2D and 3D that fixes the fragmented identities of project-and-merge pipelines, with no 3D-specific training.
Class-agnostic 3D instance segmentation is critical for robotic systems operating in unknown environments, enabling perception of previously unseen objects for reliable manipulation and navigation. Existing approaches typically project per-frame 2D instance masks into 3D and merge them, which often breaks object identities across time and yields fragmented 3D instances. We introduce Cross-Dimensional Class-Agnostic 3D Instance Segmentation (CDIS), a zero-shot framework that explicitly tracks 2D instance masks across frames and associates them with 3D superpoints, creating a feedback loop between 2D and 3D. This cross-dimensional reasoning links temporally stable 2D tracks with spatially coherent 3D regions, producing globally consistent 3D instance labels without any 3D-specific training. Experiments on benchmark datasets demonstrate that CDIS achieves higher accuracy and consistency than state-of-the-art zero-shot methods, while remaining efficient and scalable to diverse real-world environments.
CDIS produces 3D instance segmentation in three stages:
- 2D Instance Tracking — Per-frame 2D instance masks (from a class-agnostic 2D model) are made
temporally consistent by warping past frames into the current frame with the relative camera
pose and matching instances by 2D mask IoU over a sliding frame queue (
CDIS/matching_2d.py). - 3D-Guided 2D Instance Merging — Precomputed 3D superpoints are lifted against the tracked
masks and merged by 3D IoU, associating instances across frames through geometric consistency
(
CDIS/merging_3d.py,CDIS/merge_ops.py). - 3D Instance Consolidation — Duplicate assignments are resolved through 3D intersection-over-minimum and temporal co-occurrence analysis, followed by DBSCAN cleanup and intensity-weighted voxel voting, yielding a unified per-point 3D instance labelling.
Tested with Python 3.10, PyTorch 1.11.0, CUDA 11.3 on Linux with an NVIDIA GPU. See docs/INSTALL.md for the full step-by-step setup (PyTorch, pointops, detectron2 + CropFormer, mmcv). In short:
conda create -n cdis python=3.10 -y && conda activate cdis
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch -y
pip install scikit-image==0.21.0 opencv-python open3d imageio natsort numba==0.58.1
pip install git+https://github.com/facebookresearch/segment-anything.git
# then follow docs/INSTALL.md for pointops, detectron2/CropFormer, and mmcvDownload the model checkpoints into weights/ as described in weights/README.md.
CDIS is evaluated on ScanNet200. The expected data/ layout and the preprocessing steps
(posed RGB-D, mesh over-segmentation, synthetic depth, 2D masks, evaluation ground truth) are
documented in docs/DATA.md. data/ and output/ are local directories (or
symlinks) and are not tracked by git.
The pipeline runs from the repository root.
# 1. Build 3D instance predictions (tracked 2D masks -> 3D merge -> consolidation)
# bash scripts/build_CDIS.sh [CONFIG] [GPU] [PART] [SUBPROCESS_NUM]
bash scripts/build_CDIS.sh scannet200 0 0 1
# ...or split across two GPUs:
bash scripts/build_CDIS.sh scannet200 0 0 2 & # GPU 0, first half
bash scripts/build_CDIS.sh scannet200 1 1 2 & # GPU 1, second half
# 2. Evaluate class-agnostic AP / AP50 / AP25
# bash scripts/run_eval.sh [PRED_DIR] [GT_DIR]
bash scripts/run_eval.shKey hyperparameters (configs/scannet200.yaml) follow the paper:
queue_size (q_max)=5, iou_2d_th (τ_IoU2D)=0.8, iou_3d_th (τ_IoU3D)=0.6,
iom_3d_th (τ_IoMin3D)=0.8, iom_frame_th (τ_co)=0.2.
| Method | 2D Model | AP | AP50 | AP25 |
|---|---|---|---|---|
| SAM3D | SAM | 20.2 | 35.7 | 55.5 |
| Open3DIS | SAM | 31.5 | 45.3 | 51.1 |
| MaskClustering | CropFormer | 19.2 | 36.6 | 51.7 |
| OV-MAP | CropFormer | 29.9 | 49.4 | 67.5 |
| CDIS (ours) | CropFormer | 33.2 | 52.1 | 69.2 |
| CDIS (ours) | SAM | 30.1 | 44.9 | 57.8 |
Results are as reported in the paper (Table I). Run the pipeline as described under Usage to reproduce them on the ScanNet200 validation set.
CDIS is also validated qualitatively on real-world RGB-D captures (home/office scenes
reconstructed with RTAB-Map + TSDF); see configs/real_world.yaml.
CDIS/ core pipeline: run.py, matching_2d.py, merging_3d.py, merge_ops.py, segment_instances.py
utils/ data IO, 3D geometry, point-cloud / superpoint loaders
configs/ scannet200.yaml (primary), real_world.yaml, CDIS_env.yml
evaluation/ self-contained class-agnostic AP evaluator + runner
scripts/ build_CDIS.sh and data / eval preparation scripts
libs/ third-party deps: CropFormer/Entity, detectron2, pointops (not committed)
weights/ model checkpoints (downloaded, not committed)
docs/ INSTALL / DATA guides + figures
Vendored code under libs/ is adapted from the upstream projects listed under
Acknowledgements and retains their original licenses.
@inproceedings{kim2025cdis,
title = {CDIS: Cross-Dimensional Class-Agnostic 3D Instance Segmentation via 2D Mask Tracking and 3D-2D Projection Merging},
author = {Kim, Juno and Yoon, Hye-Jung and Park, Yesol and Zhang, Byoung-Tak},
booktitle = {2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
pages = {4269--4274},
year = {2025},
organization = {IEEE}
}CDIS builds on Segment Anything, CropFormer / Entity, Pointcept, SAM3D, and the ScanNet evaluation toolkit. It extends our earlier project OV-MAP. We thank the authors for releasing their code.
Released under the MIT License. Vendored third-party components remain under their respective licenses.