OPS: Open Panoramic Segmentation

Homepage

This project has been accepted by ECCV 2024! For more information about the project, please refer to our project homepage.

Prerequisites

Please make sure your CUDA is >= 11.6 since we need to compile some operators. You can use the following command to check your CUDA version:

nvcc --version

Then install all necessary packages:

conda create -n OPS python=3.9 -y
conda activate OPS
conda install pytorch==2.0.1 torchvision==0.15.2 pytorch-cuda=11.7 -c pytorch -c nvidia -y
pip install lit==18.1.8 numpy==1.23.1 cmake==3.30.4
pip install openmim==0.3.9
mim install mmengine==0.9.0
mim install mmcv==2.1.0
mim install mmsegmentation==1.2.2
pip install timm==0.9.8 einops==0.7.0 ftfy==6.1.1 pkbar==0.5 prettytable==3.9.0 py360convert==0.1.0 regex==2023.10.3 six==1.16.0
cd ops/models/dcnv3 && bash make.sh

Datasets

We train our model on COCO-Stuff164k dataset while testing on WildPASS, Matterport3D and Stanford2D3D datasets. The dataset folder structure is as follows:

OPS
├── ops
├── configs
├── pretrains
│   ├── ViT-B-16.pt
├── data
│   ├── coco_stuff164k
│   │   ├── images
│   │   │   ├── train2017
│   │   │   ├── val2017
│   │   ├── annotations
│   │   │   ├── train2017
│   │   │   ├── val2017
│   ├── matterport3d
│   │   ├── val
│   │   │   ├── rgb
│   │   │   ├── semantic
│   ├── s2d3d
│   │   ├── area1_3_6
│   │   │   ├── rgb
│   │   │   ├── semantic
│   │   ├── area2_4
│   │   │   ├── rgb
│   │   │   ├── semantic
│   │   ├── area5
│   │   │   ├── rgb
│   │   │   ├── semantic
│   ├── WildPASS
│   │   ├── images
│   │   │   ├── val
│   │   ├── annotations
│   │   │   ├── val
├── tools
├── README.md

COCO-Stuff164k

Please follow this link to download and preprocess COCO-Stuff164k dataset. As for the RERP data augmentation, please use the following command:

python tools/dataset_converters/add_erp.py --shuffle

WildPASS

Please follow WildPASS official repository to download and preprocess WildPASS dataset.

Matterport3D

Please follow 360BEV official repository to download and preprocess Matterport3D dataset.

Stanford2D3D

Please follow 360BEV official repository to download and preprocess Stanford2D3D dataset.

Pretrained CLIP

Please download the pretrained CLIP using this link. Then use tools/model_converters/clip2mmseg.py to convert model into mmseg style:

python tools/model_converters/clip2mmseg.py <MODEL_PATH> <OUTPUT_PATH> 

The processed model should be placed in pretrains folder in the end (see dataset folder structure).

Checkpoints

The checkpoints can be downloaded from:
Checkpoint without RERP
Checkpoint with RERP

Usage

Train

Please use the following command to train the model:

bash tools/dist_train.sh <CONFIG_PATH> <GPU_NUM>

<CONFIG_PATH> should be the path of the COCO_Stuff164k config file.

Test

Please use the following command to test the model:

bash tools/dist_test.sh <CONFIG_PATH> <CHECKPOINT_PATH> <GPU_NUM>

<CONFIG_PATH> should be the path of the WildPASS, Matterport3D or Stanford2D3D config file. <CHECKPOINT_PATH> should be the path of the COCO_Stuff164k checkpoint file.

Citation

If you are interested in this work, please cite as below:

@inproceedings{zheng2024open,
title={Open Panoramic Segmentation},
author={Zheng, Junwei and Liu, Ruiping and Chen, Yufan and Peng, Kunyu and Wu, Chengzhi and Yang, Kailun and Zhang, Jiaming and Stiefelhagen, Rainer},
booktitle={European Conference on Computer Vision (ECCV)},
year={2024}
}