Hiding CPU GPU synchronization latency

[mfbalin]

I have a pipeline where some operations in the pipeline are blocking because they contain operations such as copy_if or unique. I need to synchronize after these operations to transfer the length information of the output to the CPU.

I am thinking of launching these operations on the current stream asynchronously using background threads. Since the next pipeline stage can be run without having to wait for the previous stage's output (due to pipelining), I thought I can utilize the GPU better this way.

Is there anything I need to be aware of before going all-in with this approach?

[mfbalin]

It looks like the idea works.