Kimera semantics does not work with rosbag recorded from REALSENSE D435i

[lucamorando95]

Description:
Hello and congratulation for this massive framework.
It's about two days that i'm trying to use the kimera semantic framework with previously recorded rosbag data.
Kimera semantics works fine if used with euroc rosbag instead.
I am using kimera semantics with kimera Vio ROS, which works fine also on previosuly recorded rosbag.
As specified in the readme, i recorded the following topics, which are given as input to kimera VIO ROS:

Command:

/camera/infra1/image_rect_raw
/camera/infra2/image_rect_raw
/camera/imu

Here i described the launching commands pipeline:
I run kimera VIO ROS with the following command:

roslaunch kimera_vio_ros kimera_vio_ros_realsense_IR.launch 

I run kimera semantics:

roslaunch kimera_semantics_ros kimera_metric_realsense.launch 

I run kimera Rviz Vio pluging

I run kimera semantics Rviz Plugin

I run my rosbag file

while Kimera VIO ROS generates the per frame textured mesh, Kimera semantics does not create the voxblox mesh from the point cloud, differently from what it happens with the euroc dataset, where it is generated correctly.

The topic

/kimera_semantics_node/mesh

outputs mesh_block of size zero, so inside is actually empty.

The launch file kimera_vio_ros_realsense_IR is reported below:

<launch>
  <arg name="dataset_name" value="RealSenseIR"/>
  <arg name="online" default="true" />

  <arg name="log_output" default="false"/>
  <arg name="log_output_path"
       default="$(find kimera_vio_ros)/output_logs/"
       if="$(arg log_output)"/>
  <arg name="log_gt_data" default="false"/>
  <arg name="gt_topic" default=""/>
  <!-- Set use_sim_time to true if you use rosbag with clock argument -->
  <arg name="should_use_sim_time" default="false"/>

  <!-- Only used when parsing a rosbag -->
  <arg name="rosbag_path" default="/home/arpl/kimera_ws/src/rosbag/realsense_lab7.bag"
       unless="$(arg online)"/>"

  <!-- Frame IDs. These DO NOT match frame id's on the video streams, as the 
	RealSense and Kimera publish conflicting Tf's -->
  <arg name="world_frame_id"     value="world"/>
  <arg name="base_link_frame_id" value="base_link"/>
  <arg name="map_frame_id"       value="map"/>
  <arg name="left_cam_frame_id"  value="left_cam"/>
  <arg name="right_cam_frame_id" value="right_cam"/>

  <!-- Subscribed Topics -->
  <arg name="left_cam_topic"  value="/camera/infra1/image_rect_raw"/>
  <arg name="right_cam_topic" value="/camera/infra2/image_rect_raw"/>
  <arg name="imu_topic"       value="/camera/imu"/>

  <!-- Launch actual pipeline -->
  <include file="$(find kimera_vio_ros)/launch/kimera_vio_ros.launch"
           pass_all_args="true"/>

 <!-- Launch static TF node from base_link to velo_link -->
 <node pkg="tf" type="static_transform_publisher" name="velo_link_broadcaster"
   args="0 0 0 0 0 0 1 base_link velo_link 100"/>

</launch>

The launch file kimera_metric_realsense is visible below:

<?xml version="1.0" encoding="ISO-8859-15"?>
<launch>
  <!-- If you aren't running a bag file, set to false-->
  <arg name="should_use_sim_time" default="true"/>
  <arg name="max_ray_length_m"    default="2.5"/>

  <!-- If you just want to run 3D reconstruction without semantics, set this flag to false-->
  <arg name="metric_semantic_reconstruction"      value="false"/>
  <arg name="semantic_label_2_color_csv_filepath" default="$(find kimera_semantics_ros)/cfg/maskrcnn_mapping.csv"/>

  <!-- Change sensor frame to:
   - 1. VIO's estimated base_link: `left_cam_base_link`
   - 2. Or, if you want to use simulator's ground-truth: `left_cam`
  -->
  <arg name="sensor_frame" value="base_link"/>

  <!-- Input -->
  <arg name="left_cam_info_topic"         default="/depth_camera/aligned_depth_to_color/camera_info"/>
  <arg name="left_cam_topic"              default="/depth_camera/color/image_raw"/> <!--/depth_camera/color/image_raw -->
  <arg name="left_cam_segmentation_topic" default="/depth_camera/color/semantic_image"/>
  <arg name="left_cam_depth_topic"        default="/depth_camera/aligned_depth_to_color/image_raw"/>

  <!-- Tracking IMU to Tracking Camera tf -->
  <node pkg="tf" type="static_transform_publisher" name="imu_to_left_cam"
    args="-0.0107491 0.00339225 0.02843741 0.0052734748 0.0001508149  0.0022978430 0.9999834436 base_link_dvio left_cam 100"/>
  <!-- Tracking to Mapping Camera tf -->
  <node pkg="tf" type="static_transform_publisher" name="inter_cam_tf"
    args="0.075 0 -0.065 0 0.7071 0 0.7071  left_cam depth_cam 100"/>
  <node pkg="tf" type="static_transform_publisher" name="inter_cam_tf_2"
    args="0.075 0 -0.065 0 0.7071 0 0.7071 left_cam depth_camera_color_optical_frame 100"/>


  <!-- Launch actual pipeline -->
  <include file="$(find kimera_semantics_ros)/launch/kimera_semantics.launch" pass_all_args="true"/>
</launch>

Additional files:
Please attach all the files needed to reproduce the error.

Please give also the following information:

• Kimera-Semantics branch, tag or commit used
• https://github.com/MIT-SPARK/Kimera-Semantics
• GTSAM version used:
• OpenCV version used: 3.4.2
• Operating system and version (e.g. Ubuntu 16.04 or Windows 10): Ubuntu 18.04
• ROS Distribution (kinetic, melodic, …): melodic
• Did you change the source code? (yes / no): no

[CCCC-user]

hi,have you solved this problem. I hope you can give me some advice

[aswinpajayan]

From a presentation from the authors, kimera semantics will run only on EuroC (https://youtu.be/Zjevg5wQTdI?t=1001). Do you think they have added this feature now?. Refer to Issue #51

[madhu-korada]

I had to resolve all the issues related TF and it worked for me.

[prmehta24]

@madhu-korada What issues were you facing related to TF?

Did you face this problem? If so, how did you solve it? (unknown frame world)