[Questionnaire] Which model do you want to support in mmdeploy?

[VVsssssk]

Which model do you want to support in mmdeploy? Comment here. You can like a comment to increase model priority.

	Model	Codebase	Backend	Number of likes	Released
1	Swin-Transformer	mmcls	tensorrt	2	v0.6.0
2	Second	mmdet3d	tensorrt	1
3	htc	mmdet	tensorrt	1
4	Segmenter	mmseg	tensorrt	1	v0.6.0
5	ConvNeXt	mmcls	tensorrt	1
6	MASTER	mmocr	tensorrt	1
7	ABINet	mmocr	tensorrt	1
8	BEVFormer	mmdet3d	tensorrt	1
9	Oriented R-CNN	mmrotate	tensorrt	1	v0.6.0
10	SAR	mmocr	tensorrt	1

Reference:

• MASTER: ocr识别模型master // Can MASTER from mmocr be deployed?  #623
• ABINet: ABINet deployment #689
• BEVFormer: Need to deploy MultiViewImage Model BEVFormer based on mmdet3d #688
• SAR: cannot find ResizeOCR implementation on specific platform cuda #630

[VVsssssk]

E.g:
Swin-Transformer mmcls tensorrt

[OrcunCanDeniz]

second mmdet3d tensorrt

[haofanwang]

htc mmdetection tensorrt

[haofanwang]

segmenter mmsegmentation tensorrt

[starhou]

swin-transformer mmdet tensorrt

[starhou]

I haven't tried the conversion yet and I just used your example fater-rcnn. Maybe now, need to try.

[RunningLeon]

segmenter mmsegmentation tensorrt

@haofanwang Have you tried segmenter with mmdeploy?

[haofanwang]

segmenter mmsegmentation tensorrt

@haofanwang Have you tried segmenter with mmdeploy?

@RunningLeon , Yes, but I receive error messages as below

TypeError: upsample_bicubic2d() received an invalid combination of arguments - got (Tensor, NoneType, bool, list), but expected one of:
 * (Tensor input, tuple of ints output_size, bool align_corners, tuple of floats scale_factors)
      didn't match because some of the arguments have invalid types: (Tensor, NoneType, bool, list)
 * (Tensor input, tuple of ints output_size, bool align_corners, float scales_h, float scales_w, *, Tensor out)

And my command is

python ./tools/deploy.py \
    ./configs/mmseg/segmentation_tensorrt-fp16_dynamic-512x1024-2048x2048.py \
    ./mmsegmentation/configs/segmenter/segmenter_vit-l_mask_8x1_512x512_160k_ade20k.py \
    ./mmsegmentation/checkpoints/segmenter_vit-l_mask_8x1_512x512_160k_ade20k_20220105_162750-7ef345be.pth \
    ./mmsegmentation/demo/demo.png \
    --work-dir work_dir\
    --device cuda\
    --dump-in

[RunningLeon]

512x512
@haofanwang
Try with static config with 512x512. You can change in config file.

[eliabruni]

maskformer mmdet tensorrt

[isbrycee]

swin-transformer mmdet onnx

[Siannodel]

Swin-Transformer mmcls tensorrt

[shannanyinxiang]

SATRN mmocr tensorrt

[awesome01010]

SATRN for ocr, to onnx and tensorrt pls!

[summerlvsong]

SATRN for ocr, to onnx and tensorrt pls!

[Mountchicken]

SATRN MMOCR tensorrt

[Xiangxu-0103]

dynamic_voxelization mmdet3d onnxruntime and tensorrt

[YongboLiang]

Swin-Transformer Second Segmenter

[hust-lidelong]

Oriented R-CNN，S2A-Net for mmrotate tensorrt, please!

[eliabruni]

mask2former mmdet tensorrt

[jianghaijun007]

monoflex tensorrt

[gzxy-0102]

mmtracking

[troyliu0105]

AssociativeEmbedding pose detection in mmpose

[SpaceView]

mmocr

[Zaughter]

dab-detr onnx

[RunningLeon]

dab-detr onnx

@Zaughter @DuckJ @sibet-lb @RobinHan24 @idonahum1
hi, for DETR, DINO, Dab-DETR, Conditional-DETR, could try this PR##2410

[RobinHan24]

dab-detr onnx

@Zaughter @DuckJ @sibet-lb @RobinHan24 @idonahum1 hi, for DETR, DINO, Dab-DETR, Conditional-DETR, could try this PR##2410

I just tried it, but error.

2023-09-10:10:53:30 - root - ERROR - execute_async_v2(): incompatible function arguments. The following argument types are supported:
1. (self: tensorrt.tensorrt.IExecutionContext, bindings: List[int], stream_handle: int, input_consumed: capsule = None) -> bool

Invoked with: <tensorrt.tensorrt.IExecutionContext object at 0x7f01857b5bb0>, [139631165177856, None, 139634709371904, 139634709365760], 0
Traceback (most recent call last):
File "/mnt/sda/qilibin/project/mmdeploy-support_detr_multi-batch/mmdeploy/utils/utils.py", line 41, in target_wrapper
result = target(*args, **kwargs)
File "/mnt/sda/qilibin/project/mmdeploy-support_detr_multi-batch/mmdeploy/apis/visualize.py", line 72, in visualize_model
result = model.test_step(model_inputs)[0]
File "/mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/mmengine/model/base_model/base_model.py", line 145, in test_step
return self._run_forward(data, mode='predict') # type: ignore
File "/mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/mmengine/model/base_model/base_model.py", line 340, in _run_forward
results = self(**data, mode=mode)
File "/mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/sda/qilibin/project/mmdeploy-support_detr_multi-batch/mmdeploy/codebase/mmdet/deploy/object_detection_model.py", line 299, in forward
outputs = self.predict(inputs, shape_info)
File "/mnt/sda/qilibin/project/mmdeploy-support_detr_multi-batch/mmdeploy/codebase/mmdet/deploy/object_detection_model.py", line 323, in predict
outputs = self.wrapper(inputs)
File "/mnt/sda/qilibin/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/mnt/sda/qilibin/project/mmdeploy-support_detr_multi-batch/mmdeploy/backend/tensorrt/wrapper.py", line 174, in forward
self.__trt_execute(bindings=bindings)
File "/mnt/sda/qilibin/project/mmdeploy-support_detr_multi-batch/mmdeploy/utils/timer.py", line 67, in fun
result = func(*args, **kwargs)
File "/mnt/sda/qilibin/project/mmdeploy-support_detr_multi-batch/mmdeploy/backend/tensorrt/wrapper.py", line 185, in __trt_execute
self.context.execute_async_v2(bindings,
TypeError: execute_async_v2(): incompatible function arguments. The following argument types are supported:
1. (self: tensorrt.tensorrt.IExecutionContext, bindings: List[int], stream_handle: int, input_consumed: capsule = None) -> bool

Invoked with: <tensorrt.tensorrt.IExecutionContext object at 0x7f01857b5bb0>, [139631165177856, None, 139634709371904, 139634709365760], 0
09/10 10:53:32 - mmengine - ERROR - /mnt/sda/qilibin/project/mmdeploy-support_detr_multi-batch/tools/deploy.py - create_process - 82 - visualize tensorrt model failed.

tensorrt version: 8.6.1.6

command:
python tools/deploy.py configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py dino/dino-4scale_r50_8xb2-36e_coco.py dino-4scale_r50_8xb2-12e_coco_20221202_182705-55b2bba2.pth demo/resources/det.jpg --work-dir mmdeploy_models/mmdet/dino --device cuda --show --dump-info

could you please help me？

[RobinHan24]

dab-detr onnx

@Zaughter @DuckJ @sibet-lb @RobinHan24 @idonahum1 hi, for DETR, DINO, Dab-DETR, Conditional-DETR, could try this PR##2410
thanks a lot, I have got the tensorrt model. But how to infer with trt model sdk?
my code:
from mmdeploy_runtime import Detector
import cv2
img = cv2.imread('./demo/resources/det.jpg')
detector = Detector(model_path='mmdeploy_models/mmdet/dino0910', device_name='cuda', device_id=0)
res = detector(img)
print (res)
indices = [i for i in range(len(bboxes))]
for index, bbox, label_id in zip(indices, bboxes, labels):
[left, top, right, bottom], score = bbox[0:4].astype(int), bbox[4]
if score < 0.3:
continue
cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))
cv2.imwrite('output_detection.png', img)

error:
[2023-09-10 12:51:09.788] [mmdeploy] [error] [net_module.cpp:130] sample 0 missing key shape

can you help me, thanks.

[MarouaneMja]

CO-DETR https://github.com/Sense-X/Co-DETR -> onnx and tensorrt

[x-yy0]

mvxnet mmdet3d tensorrt

[mjehanzaib999]

slowonly(spatiotemporal) mmaction tensorrt

[DoctorDream]

Co-DETR please!

[Mayyyybe]

Co-DETR to onnx please！！！

[MarouaneMja]

@DoctorDream @Mayyyybe I already mentionned CO-DETR, Could you please upvote my suggestion , so that it could increase this model priority?

[Baboom-l]

Grounding DINO mmdetection tensorrt

[xinlin-xiao]

CO-DETR https://github.com/Sense-X/Co-DETR -> onnx and tensorrt

CO-DETR->onnx ,TorchScript,tensorrt

[OSSome01]

KNet mmsegmentaion
ConvNext mmsegmentation

[WaseemDR]

TensorRT for Grounding Dino

[Jialeen]

TensorRT for Detic

[iceriver97]

fcos3d mmdet3d tensorrt

这个目前实现了吗？

[chunxt]

tood. thanks

[Nieees]

htc Series models （scnet， detectRS） Is there any progress so far?

[Bovey0809]

MMRazor QAT

[monologuesmw]

TOOD model. Thanks!

[monkeycc]

YOLOv8
[Detect]
[Segment]
[Classify]
[Pose]
[OBB]

[ZJU-lishuang]

glip | mmdet | tensorrt

[chenzhutian]

3D pose (eg. MotionBert) in mmPose

[xlg-go]

SVTR for ocr, to onnx and tensorRT pls!

[MarouaneMja]

CO-DETR https://github.com/Sense-X/Co-DETR -> onnx and tensorrt

Hello @VVsssssk ,

Any updates about the exportation of Co-DETR to onnx and tensorrt please?

Best regards,
Marouane

[songkemail]

superpoint superglue

[sjtu-cz]

Swin-Transformer v2 mmpretrain tensorRT

[wostons]

grounding_dino

[Liufanfanfan]

ddrnet-slim mmseg