ROS_NCNN

This is a ROS package for NCNN, a high-performance neural network inference framework - by Tencent - optimized for mobile platforms:

• ARM NEON assembly level optimization
• Sophisticated memory management and data structure design, very low memory footprint
• Supports multi-core parallel computing acceleration
• Supports GPU acceleration via the next-generation low-overhead Vulkan API
• The overall library size is less than 700K, and can be easily reduced to less than 300K
• Extensible model design, supports 8bit quantization and half-precision floating point storage
• Can import caffe/pytorch/mxnet/onnx models

Setting up

Library

• Build for NVIDIA Jetson
• Build for Linux x86
• Build for Windows x64 using VS2017
• Build for MacOSX
• Build for Raspberry Pi 3
• Build for ARM Cortex-A family with cross-compiling
• Build for Android
• Build for iOS on MacOSX with xcode
• Build for iOS on Linux with cctools-port
• Build for Hisilicon platform with cross-compiling

ROS package

• Clone this repository into your catkin workspace.
• Initialize and update submodule ncnn-assets ( this is a collection of some popular models )
• Compile the workspace.
• CMake script is going to autodetect whether the ncnn library is built with Vulkan or not. ( All nodes will utilize the GPU if Vulkan is enabled. )

General launch parameters

<node name="yolact_node" pkg="ros_ncnn" type="yolact_node" output="screen">
  <param name="display_output" value="$(arg display_output)"/>
  <remap from="/camera/image_raw" to="$(arg camera_topic)"/>
  <!-- Select discrete GPU, in any other case the node jumps to the first discrete GPU. -->
  <param name="gpu_device" value="0"/>
  <!-- Number of CPU threads to use, uses all available if not provided. -->
  <param name="num_threads" value="8"/>
  <!-- Turns engine.neuralnet.opt.use_vulkan_compute regardless the lib is built with GPU support -->
  <param name="enable_gpu" value="true"/>
</node>

YOLACT

Publisher

# Object message
Header header
Rectangle boundingbox # Vector2D position and size
string label
float32 probability

Params

• probability_threshold - default 0.5 - above which objects are published

YOLO v2 / v3

The assets repository has multiple YOLO networks, choose the parameter and model file before launch. ( Default is YOLO-3 on MobileNet-2 )

Publisher

# Object message
Header header
Rectangle boundingbox # Vector2D position and size
string label
float32 probability

Params

• model_file - YOLO network model file
• param_file - YOLO network parameter file
• probability_threshold - default 0.5 - above which objects are published

YOLO v5

Publisher

# Object message
Header header
Rectangle boundingbox # Vector2D position and size
string label
float32 probability

Params

• model_file - YOLO network model file
• param_file - YOLO network parameter file
• probability_threshold - default 0.5 - above which objects are published

RetinaFace

Publisher

# FaceObject message
Header header
Rectangle boundingbox # Vector2D position and size
Vector2D[5] landmark # 5x 2x float32
float32 probability

Params

• probability_threshold - default 0.5 - above which face objects are published

HopeNet

Using RetinaFace as face detector:

Publisher

# Euler angles
float32 roll
float32 pitch
float32 yaw

PoseNet

Faster R-CNN

Don't forget to uncompress ZF_faster_rcnn_final.bin.zip in assets directory first. ( but again, R-CNN is the past and that's neither a cat nor a bird right there... that's my best friend )

🚧 To do

• General model loader node ( with layer to topic mapping through NDS file )
• Dynamic reconfiguration for some params ( e.g. probability thresholds )

✌️ Acknowledgements

Special thanks to Nihui for her wonderful work.