Skip to content

Latest commit

 

History

History
185 lines (134 loc) · 13.3 KB

README.md

File metadata and controls

185 lines (134 loc) · 13.3 KB
Raw

MapTR

An End-to-End Framework for Online Vectorized HD Map Construction

Bencheng Liao1,2,3 *, Shaoyu Chen1,3 *, Yunchi Zhang1,3 , Bo Jiang1,3 ,Tianheng Cheng1,3, Qian Zhang3, Wenyu Liu1, Chang Huang3, Xinggang Wang1 📧

1 School of EIC, HUST, 2 Institute of Artificial Intelligence, HUST, 3 Horizon Robotics

(*) equal contribution, (📧) corresponding author.

ArXiv Preprint (arXiv 2208.14437)

openreview ICLR'23, accepted as ICLR Spotlight

extended ArXiv Preprint MapTRv2 (arXiv 2308.05736), accepted to IJCV 2024

News

  • Oct. 6th, 2024: MapTRv2 is accepted to IJCV 2024!
  • Feb 20th, 2024: MapTRv2-based VADv2 is presented on arXiv paper project page.
  • Aug. 31th, 2023: initial MapTRv2 is released at maptrv2 branch. Please run git checkout maptrv2 to use it.
  • Aug. 14th, 2023: As required by many researchers, the code of MapTR-based map annotation framework (VMA) will be released at https://github.com/hustvl/VMA recently.
  • Aug. 10th, 2023: We release MapTRv2 on Arxiv. MapTRv2 demonstrates much stronger performance and much faster convergence. To better meet the requirement of the downstream planner (like PDM), we introduce an extra semantic——centerline (using path-wise modeling proposed by LaneGAP). Code & model will be released in late August. Please stay tuned!
  • May. 12th, 2023: MapTR now support various bevencoder, such as BEVFormer encoder and BEVFusion bevpool. Check it out!
  • Apr. 20th, 2023: Extending MapTR to a general map annotation framework (paper, code), with high flexibility in terms of spatial scale and element type.
  • Mar. 22nd, 2023: By leveraging MapTR, VAD (paper, code) models the driving scene as fully vectorized representation, achieving SoTA end-to-end planning performance!
  • Jan. 21st, 2023: MapTR is accepted to ICLR 2023 as Spotlight Presentation!
  • Nov. 11st, 2022: We release an initial version of MapTR.
  • Aug. 31st, 2022: We released our paper on Arxiv. Code/Models are coming soon. Please stay tuned! ☕️

Introduction

MapTR/MapTRv2 is a simple, fast and strong online vectorized HD map construction framework.

framework

High-definition (HD) map provides abundant and precise static environmental information of the driving scene, serving as a fundamental and indispensable component for planning in autonomous driving system. In this paper, we present Map TRansformer, an end-to-end framework for online vectorized HD map construction. We propose a unified permutation-equivalent modeling approach, i.e., modeling map element as a point set with a group of equivalent permutations, which accurately describes the shape of map element and stabilizes the learning process. We design a hierarchical query embedding scheme to flexibly encode structured map information and perform hierarchical bipartite matching for map element learning. To speed up convergence, we further introduce auxiliary one-to-many matching and dense supervision. The proposed method well copes with various map elements with arbitrary shapes. It runs at real-time inference speed and achieves state-of-the-art performance on both nuScenes and Argoverse2 datasets. Abundant qualitative results show stable and robust map construction quality in complex and various driving scenes.

Models

Results from the MapTRv2 paper

comparison

Method Backbone Lr Schd mAP FPS
MapTR R18 110ep 45.9 35.0
MapTR R50 24ep 50.3 15.1
MapTR R50 110ep 58.7 15.1
MapTRv2 R18 110ep 52.3 33.7
MapTRv2 R50 24ep 61.5 14.1
MapTRv2 R50 110ep 68.7 14.1
MapTRv2 V2-99 110ep 73.4 9.9

Notes:

  • FPS is measured on NVIDIA RTX3090 GPU with batch size of 1 (containing 6 view images).
  • All the experiments are performed on 8 NVIDIA GeForce RTX 3090 GPUs.

Results from this repo.

MapTR

nuScenes dataset

Method Backbone BEVEncoder Lr Schd mAP FPS memory Config Download
MapTR-nano R18 GKT 110ep 46.3 35.0 11907M (bs 24) config model / log
MapTR-tiny R50 GKT 24ep 50.0 15.1 10287M (bs 4) config model / log
MapTR-tiny R50 GKT 110ep 59.3 15.1 10287M (bs 4) config model / log
MapTR-tiny Camera & LiDAR GKT 24ep 62.7 6.0 11858M (bs 4) config model / log
MapTR-tiny R50 bevpool 24ep 50.1 14.7 9817M (bs 4) config model / log
MapTR-tiny R50 bevformer 24ep 48.7 15.0 10219M (bs 4) config model / log
MapTR-tiny+ R50 GKT 24ep 51.3 15.1 15158M (bs 4) config model / log
MapTR-tiny+ R50 bevformer 24ep 53.3 15.0 15087M (bs 4) config model / log

Notes:

  • + means that we introduce temporal setting.

MapTRv2

Please git checkout maptrv2 and follow the install instruction to use following checkpoint

nuScenes dataset

Method Backbone BEVEncoder Lr Schd mAP FPS memory Config Download
MapTRv2 R50 bevpool 24ep 61.4 14.1 19426M (bs 24) config model / log
MapTRv2* R50 bevpool 24ep 54.3 WIP 20363M (bs 24) config model / log

Argoverse2 dataset

Method Backbone BEVEncoder Lr Schd mAP FPS memory Config Download
MapTRv2 R50 bevpool 6ep 64.3 14.1 20580 (bs 24) config model / log
MapTRv2* R50 bevpool 6ep 61.3 WIP 21515 (bs 24) config model / log

Notes:

  • * means that we introduce an extra semantic——centerline (using path-wise modeling proposed by LaneGAP).

Qualitative results on nuScenes val split and Argoverse2 val split

MapTR/MapTRv2 maintains stable and robust map construction quality in various driving scenes.

visualization

MapTRv2 on whole nuScenes val split

Youtube

MapTRv2 on whole Argoverse2 val split

Youtube

End-to-end Planning based on MapTR

e2e_planning.mp4

Getting Started

Catalog

  • temporal modules
  • centerline detection & topology support (refer to maptrv2 branch)
  • multi-modal checkpoints
  • multi-modal code
  • lidar modality code
  • argoverse2 dataset
  • Nuscenes dataset
  • MapTR checkpoints
  • MapTR code
  • Initialization

Acknowledgements

MapTR is based on mmdetection3d. It is also greatly inspired by the following outstanding contributions to the open-source community: BEVFusion, BEVFormer, HDMapNet, GKT, VectorMapNet.

Citation

If you find MapTR is useful in your research or applications, please consider giving us a star 🌟 and citing it by the following BibTeX entry.

@article{liao2024maptrv2,
  title={Maptrv2: An end-to-end framework for online vectorized hd map construction},
  author={Liao, Bencheng and Chen, Shaoyu and Zhang, Yunchi and Jiang, Bo and Zhang, Qian and Liu, Wenyu and Huang, Chang and Wang, Xinggang},
  journal={International Journal of Computer Vision},
  pages={1--23},
  year={2024},
  publisher={Springer}
}
@inproceedings{MapTR,
  title={MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction},
  author={Liao, Bencheng and Chen, Shaoyu and Wang, Xinggang and Cheng, Tianheng, and Zhang, Qian and Liu, Wenyu and Huang, Chang},
  booktitle={International Conference on Learning Representations},
  year={2023}
}
@inproceedings{liao2025lane,
  title={Lane graph as path: Continuity-preserving path-wise modeling for online lane graph construction},
  author={Liao, Bencheng and Chen, Shaoyu and Jiang, Bo and Cheng, Tianheng and Zhang, Qian and Liu, Wenyu and Huang, Chang and Wang, Xinggang},
  booktitle={European Conference on Computer Vision},
  pages={334--351},
  year={2024},
  organization={Springer}
}