CoMAE

[AAAI 2023 Oral] CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets

Prepare Data

Baiduyun(https://pan.baidu.com/s/1LZIF1hlT3k0oX76Ttp660w) The extraction code is: g5vp

Dependencies

• python 3.7.4
• torch 1.7.0
• torchvision 0.8.1
• timm 0.3.2
• numpy 1.17.2

Pre-training

Note give your own data_path, output_dir and log_dir in command parameters.

python main_pretrain_cpc.py or

python -m torch.distributed.launch --nproc_per_node NUM_GPU main_pretrain_cpc.py

Load CPC pretrained weights and python main_pretrain_mm_mae.py or

python -m torch.distributed.launch --nproc_per_node NUM_GPU main_pretrain_mm_mae.py

Fine-tuning and Evaluating

Note give your own data_path, output_dir, log_dir and finetune in command parameters.

python main_finetune.py or

python -m torch.distributed.launch --nproc_per_node NUM_GPU main_finetune.py

Checkpoints on SUN RGB-D

cpc_stage1 Google Drive

mm_mae_stage2 Google Drive

finetune Google Drive

Citation

Please cite the following paper if you feel this repository useful for your research.

@article{yang2023comae,
  title={CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets},
  author={Yang, Jiange and Guo, Sheng and Wu, Gangshan and Wang, Limin},
  journal={arXiv preprint arXiv:2302.06148},
  year={2023}
}


@inproceedings{DBLP:conf/aaai/Yang0W023,
  author       = {Jiange Yang and
                  Sheng Guo and
                  Gangshan Wu and
                  Limin Wang},
  editor       = {Brian Williams and
                  Yiling Chen and
                  Jennifer Neville},
  title        = {CoMAE: Single Model Hybrid Pre-training on Small-Scale {RGB-D} Datasets},
  booktitle    = {Thirty-Seventh {AAAI} Conference on Artificial Intelligence, {AAAI}
                  2023, Thirty-Fifth Conference on Innovative Applications of Artificial
                  Intelligence, {IAAI} 2023, Thirteenth Symposium on Educational Advances
                  in Artificial Intelligence, {EAAI} 2023, Washington, DC, USA, February
                  7-14, 2023},
  pages        = {3145--3154},
  publisher    = {{AAAI} Press},
  year         = {2023},
  url          = {https://doi.org/10.1609/aaai.v37i3.25419},
  doi          = {10.1609/AAAI.V37I3.25419},
  timestamp    = {Mon, 04 Sep 2023 16:50:28 +0200},
  biburl       = {https://dblp.org/rec/conf/aaai/Yang0W023.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

Acknowledges

This repo contains modified codes from: MAE.