Segmentation labels/masks from depth camera image

[k-maheshkumar]

I see that there is segmentaion camera, that ouputs labels/masks for the instance and semantic segmentation. This is based on the 2D images, Is there any way to get 2D masks from the depth camera output and not setting up a seperate camera?

Desired behavior

1. Add rgb/depth camera into simulation
2. publish segmentation labels/masks from the above camera (incase of depth camera, labels on the depth image) instead of adding new segmentation camera

Alternatives considered

If there is already a solution or a workaround, I would be happy to know about it :)

[k-maheshkumar]

similarly, it would be nicer to have the same behaviour for other camera sensors

[iche033]

This is currently not possible. Another option would be to extend the segmentation camera to output depth images with labels but that would also require some work. The easiest workaround is to set up 2 separate sensors for now.

[k-maheshkumar]

sorry, it was a typo, I meant 2D masks. In other words, can we integrate depth camera and segmentation camera?

[mjcarroll]

I believe that if you place two cameras, a segmentation and a depth camera, in the same location with the same parameters (focal length and resolution), the results may be what you are looking for. This is essentially how the RGB-Depth camera works as well.

[k-maheshkumar]

Thnaks for the suggestion. I have tried the same and got the expected output, but are those cameras time synchronized?.

Also I have some problems (see below), you can refer to the attached sdf example and the screenshot.

Scenario: spawned few objects and three set of camera (each set has monochrome, depth and segmentation camera)
System specs: AMD Ryzen 7 5800X, 16 GB RAM, NVIDIA GeForce RTX 3070 Ti
Gazebo version: Fortress (with real_time_update_rate set to 100)

Problems:
Looking into the source code, I know, when the cameras are not subscribed, nothing happens in the background, but

• Whenever the cameras are visualized, the real time factor (RTF) drops
• more camera visualized, more RTF drops, this is more significant when monochrome image is subscribed

Could please let me know if miss something or is it a known issue or got solved in the newest version already?

Also I have tried to create a custom plugin and found that ConnectNewImageFrame (for camera) callback is never triggered similar to the callback from ConnectNewDepthFrame (for depth camera). Is there a way that we get the callback triggered, so I have an access to the raw buffer?

Attachments:
cameras.sdf.txt