Setup Orin & ZED Cameras

[jruths]

The new NVIDIA Orin needs to be set up and configured to aggregate images & depth images from the four ZED cameras. Stereolabs (ZED) has a ROS SDK for doing much of the calibration and aggregation from multiple cameras. We should be able to stitch together a birds-eye-view image (often used throughout the industry).

In terms of design, the Orin will receive all the ZED data - and possibly also the LIDAR and RADAR data, do any needed pre-processing (e.g., birds-eye-view image, ground plane removal for lidar, etc) and publish cleaned perception topics to the main computer. To do this, the Orin and main computer will need to be on the same local network to share ROS messages.

This task could be coordinated with a member of the HFE team or done entirely by one person. Either way there will be a lot of hands on.

[jruths]

Some related tasks mainly since the ZED SDK has built in ways to configure the 4 ZED cameras into a single 360-degree image and depth map. Then this depth map could be used and aligned with the LIDAR and RADAR point clouds to determine the transformations that align these with the depth map.

• Automatic Sensor Calibration
• Projecting images onto the pointcloud

[MtGuerenS]

09/09/2024 Update:

Status: Setting up the NVIDIA Jetson ORIN

Next Steps:

1. Setup Nvidia ORIN
   a. Connect the physical
   b. Follow the tutorial
2. Complete the setup for ZED cameras
3. Test out ZED cameras on ORIN
4. Download docker on ORIN
5. Setup ROS2 and test it
6. Connect ORIN to the vehicle local network
7. Test ROS2 messaging across the devices
8. Setup the ROS2 ZED nodes and messages
9. Mount the ZED cameras and ORIN

Update:

• Acquired the necessary components for ORIN
• Notice factory reset needed -- past user logged in
• Fixed a one-line syntax issue with dev branch and pushed out change to dev
• Helped to fix the traffic light detector -- missing model file + dependency issues

[MtGuerenS]

09/16/2024 Update:

Status: Setting up the NVIDIA Jetson ORIN

Next Steps:

1. Setup Nvidia ORIN
   a. Connect the physical
   b. Factory reset the device
   c. Install JetPack Components
2. Complete the setup for ZED cameras
3. Test out ZED cameras on ORIN
4. Download docker on ORIN
5. Setup ROS2 and test it
6. Connect ORIN to the vehicle local network
7. Test ROS2 messaging across the devices
8. Setup the ROS2 ZED nodes and messages
9. Mount the ZED cameras and ORIN

Update:

• Borrowed a computer from Justin that had Ubuntu 22.04
• Redownloaded Ubuntu 22.04 onto the computer
• Downloaded SDK Manager for flashing (resetting) the NVIDIA ORIN
• Follow the instructions to flash the ORIN
• Setup Ubuntu on the device

[MtGuerenS]

09/23/2024 Update:

Status: Setting up the NVIDIA Jetson ORIN

Next Steps:

1. Setup Nvidia ORIN
   a. Connect the physical
   b. Factory reset the device
   c. Install JetPack Components
2. Complete the setup for ZED cameras
3. Test out ZED cameras on ORIN
4. Download docker on ORIN
5. Setup ROS2 and test it
6. Connect ORIN to the vehicle local network
7. Test ROS2 messaging across the devices
8. Setup the ROS2 ZED nodes and messages
9. Mount the ZED cameras and ORIN

Update:

• Connected the GMSL2 Capture to the ORIN
• Downloaded the ZED Link SDK for the GMSL2 connection
• Downloaded Docker and setup a docker container for ZED SDK
• Running into error with camera connection

[MtGuerenS]

09/31/2024 Update:

Status: Completed ZED Setup and Testing on Orin

Next Steps:

1. Setup Nvidia ORIN
   a. Connect the physical
   b. Factory reset the device
   c. Install JetPack Components
2. Complete the setup for ZED cameras
3. Test out ZED cameras on ORIN
4. Download docker on ORIN
5. Setup ROS2 and test it
6. Connect ORIN to the vehicle local network
7. Test ROS2 messaging across the devices
8. Setup the ROS2 ZED nodes and messages
9. Mount the ZED cameras and ORIN

Update:

• Downloaded the ZED SDK on the host machine
• Tested ZED Tools and temporarily mounted cameras with zip ties
• Figured out only two cameras can be connected to 2/4 of the connections for the ZED Quad Link (4 Cameras avaiable total)
• Re-attempted to setup the ZED SDK in a Docker container
  • Debugged for a while and ran into some permission issues with nvidia/gmsl in docker

[MtGuerenS]

10/28/2024 Update:

Status: Connecting ZED and ROS

Next Steps:

1. Setup Nvidia ORIN
   a. Connect the physical
   b. Factory reset the device
   c. Install JetPack Components
2. Complete the setup for ZED cameras
3. Test out ZED cameras on ORIN
4. Download docker on ORIN
5. Setup/Configure ROS2
6. Connect/Test ROS2 and ZED SDK
7. Connect ORIN to the vehicle local network
8. Test ROS2 messaging across the devices
9. Setup the ROS2 ZED nodes and messages
10. Mount the ZED cameras and ORIN

Update:

• Followed up with Stereo Labs community issue post
• Downloaded and setup ROS2 and ZEDX ROS2 Wrapper
• Ran into issue with optimizing Neural Depth Plus model

[MtGuerenS]

11/4/2024 Update:

Status: Connecting ZED and ROS

Next Steps:

1. Setup Nvidia ORIN
   a. Connect the physical
   b. Factory reset the device
   c. Install JetPack Components
2. Complete the setup for ZED cameras
3. Test out ZED cameras on ORIN
4. Download docker on ORIN
5. Setup/Configure ROS2
6. Connect/Test ROS2 and ZED SDK
7. Connect ORIN to the vehicle local network
8. Test ROS2 messaging across the devices
9. Setup the ROS2 ZED nodes and messages
10. Mount the ZED cameras and ORIN

Update:

• Followed up with Stereo Labs community issue post
• Successfully ran zed_camera.launch.py on ros2 with over 10 Hz
  • Need to fully measure speed
• Figured out how to control the settings for running each camera
• Need to test running multiple cameras