Autolabeling 3D Objects With Differentiable Rendering of SDF Shape Priors

Official PyTorch implementation of the CVPR 2020 paper "Autolabeling 3D Objects With Differentiable Rendering of SDF Shape Priors" by the ML Team at Toyota Research Institute (TRI), cf. References below. [Full paper] [YouTube]

Setting up your environment

To set up the environment using conda, use the following commands:

conda env create -n sdflabel -f environment.yml
conda activate sdflabel

Add the sdfrenderer directory to PYTHONPATH:

export PYTHONPATH="${PYTHONPATH}:/path/to/sdfrenderer"

Optimization demo

To run the optimization demo, first download the data folder. Then, extract the archive to the root folder of the project and run the following command:

python main.py configs/config_refine.ini --demo

Training CSS network

To train the CSS network, run the following command:

python main.py configs/config_train.ini --train

Dataset format

The dataset of crops represents a collection of detected RGB patches (CSS input), corresponding NOCS patches (CSS output), and a JSON DB file comprising the patch relevant information (most importantly SDF latent vectors corresponding to the depicted 3D models). An example of such dataset is located in the data/db/crops folder.

Optimization

Download KITTI 3D and modify the kitti_path in the config file config_refine.ini accordingly. To run optimization on the KITTI 3D dataset, run the following command:

python main.py configs/config_refine.ini --refine

Upon completion, autolabels will be stored to the output folder specified in the config file (output -> labels). To evaluate the generated dump, run:

python main.py configs/config_refine.ini --evaluate

License

The source code is released under the MIT license.

References

Autolabeling 3D Objects With Differentiable Rendering of SDF Shape Priors (CVPR 2020 oral)

Sergey Zakharov*, Wadim Kehl*, Arjun Bhargava, Adrien Gaidon

@inproceedings{sdflabel,
author = {Sergey Zakharov and Wadim Kehl and Arjun Bhargava and Adrien Gaidon},
title = {Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors},
booktitle = {IEEE Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}