Can point cloud upsampling downsampling be done during network forward pass?

[liuzhy71]

Greate work in point cloud processing!

I noticed that in your work, point cloud sampling and neighbor search is done before the data is fed into the network. I wonder why this should be done in advance not during the forward pass of the network? And what is the drawbacks if the subsampling and the KNN is done during the forward pass?

[HuguesTHOMAS]

Hi @liuzhy71,

If you downsample during the forward path, then you also have to compute convolution neighbors during the forward path. If downsampling is super fast, the neighbor computation is not so much. Therefore, some non-negligible amount of time will be used for downsampling + neighbor computation, slowing everything down.

If you downsample before the forward path, you can actually use your CPU efficiently and compute downsamping + neighbors WHILE the GPU is computing the previous forward path. Therefore the forward path is faster, and when it is done, a new input is already ready with downsampling and neighbors already computed. This is what we do here.

[liuzhy71]

If my understanding is right, proprocessing the point cloud before the forward pass is used to speed up the training time. While for the inference time, the total amount of time is still the same. And furthermore, if I want to have a dynamic point cloud hierarchy of different resolution, the subsampling and neighbor search should be inserted into the forward pass?

[HuguesTHOMAS]

In the case of inference, the processing time is also reduced. We can achieve a higher processing frequency for offline inference tasks. For online inference in real time, our pipeline is not ideal and has big delays.

Yes for dynamic point cloud hierarchy, you have to subsample during forward pass. A new version of the code will soon come out with this option.

In the mean time you can have a look at https://github.com/qinzheng93/Easy-KPConv which solves this