How to run on PC / Desktop

[SrinivasuluCharupally]

Hi,
I am trying to setup it on PC / Desktop, Intel CPU + Ubuntu 20.04.
I followed the install steps from here : https://github.com/flowdriveai/flowpilot/wiki/Installation#installing-on-desktop
Then I followed "Running on Simulator" steps from here : https://github.com/flowdriveai/flowpilot/wiki/Running-On-Simulation#following-steps-will-help-you-get-carla-simulation---flowpilot-integration-up-and-running

I got confused with these following two steps... Could you help to clarigy

• start carla simulation using ./CarlaUE4.sh -RenderOffScreen
• In one terminal, launch flowpilot and in other, launch python tool/sim/bridge.py

This means, total 3 terminals do i need to launch, one for "./CarlaUE4.sh -RenderOffScreen", one for "./launch_flowpilot.sh" and one for "python tool/sim/bridge.py"

Is that correct ?
and, Do i need to have Camera or SmartPhone to run the application ?

[MankaranSingh]

Yes, you need to run them parallely in 3 seperate terminals.

[SrinivasuluCharupally]

Thanks Mankaran. One more thing.. This is what i am getting when try to run .
I am running from SSH.

`(flowpilot) che7@che7-w30871:~/flowpilot$ ./CarlaUE4.sh -RenderOffScreen
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.

che7@che7-w30871:~/flowpilot$ ./launch_flowpilot.sh
logmessaged thermald_ statsd keyvald flowpilot pandad deleter flowinitd
logmessaged thermald_ statsd keyvald flowpilot pandad deleter flowinitd
logmessaged thermald_ statsd keyvald flowpilot pandad deleter flowinitd
logmessaged thermald_ statsd keyvald flowpilot pandad deleter flowinitd
accepted_terms False
completed_training False
ignition False

che7@che7-w30871:~/flowpilot$ python3 tools/sim/bridge.py
Restarting bridge. Error: time-out of 5000ms while waiting for the simulator, make sure the simulator is ready and connected to 127.0.0.1:2000
Restarting bridge. Error: time-out of 5000ms while waiting for the simulator, make sure the simulator is ready and connected to 127.0.0.1:2000
Restarting bridge. Error: time-out of 5000ms while waiting for the simulator, make sure the simulator is ready and connected to 127.0.0.1:2000

`

[RaghavPrabhakar66]

(flowpilot) che7@che7-w30871:~/flowpilot$ ./CarlaUE4.sh -RenderOffScreen
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.

Is Carla running properly?

[SrinivasuluCharupally]

Not sure, doing it for the first time.
i running it from SSH.

[MankaranSingh]

You will have to make sure carla is running. If you remove the RenderOffScreen arg, it should show the carla window which will confirm its up and running

[SrinivasuluCharupally]

Could you please confirm this error. this is OK or, Not ?

(flowpilot) che7@che7-w30871:~/flowpilot$ python scripts/dummy_params.py
:228: RuntimeWarning: compiletime version 3.8 of module 'common.params_pyx' does not match runtime version 3.9
:228: RuntimeWarning: compiletime version 3.8 of module 'common.transformations.transformations' does not match runtime version 3.9
done

[MankaranSingh]

I think you compiled (ran scons command) without entering the flowpilot env (pip env shell)
Do pip env shell then scons in flowpilot root directory then this warning should go away

[SrinivasuluCharupally]

Got it, thanks Mankaran.

[SrinivasuluCharupally]

MankaranSingh - We tried the steps but that did not work for us. Attaching a screenRecording of what we did. Could tell us what was the issue ?

jFYI ... Prithiveen from our team tried the steps.

Untitled.1.mp4

https://github.com/PrithiveenKumaarRamkumar

[RaghavPrabhakar66]

@SrinivasuluCharupally Can you share the output of following command.

file tmp.avi

[SrinivasuluCharupally]

@RaghavPrabhakar66
che7@che7-w30871:~/flowpilot/selfdrive/assets$ file tmp
tmp: RIFF (little-endian) data, AVI, 1164 x 874, 25.00 fps, video: Motion JPEG

[PrithiveenKumaarRamkumar]

@MankaranSingh @RaghavPrabhakar66
Able to successfully launch flowpilot app now :)
bridge.py still renders a Segmentation fault (core dumped) error

flowpilot_app_journey.mp4

[MankaranSingh]

I'd recommed you clean the build using scons -c.
I am not sure but maybe when you interchanged the python version (3.10) and ran scons that built some files that were not compatible.

[SrinivasuluCharupally]

@MankaranSingh , @RaghavPrabhakar66 - Could we have short call some time next week to look into the issues ?
We are from Valeo team / org , trying to explore the option for different platform.

[PrithiveenKumaarRamkumar]

@MankaranSingh @RaghavPrabhakar66
Tired cleaning the build using scons -c and then building again using scons but bridge.py still renders a Segmentation fault (core dumped) error
user journey: https://drive.google.com/file/d/1h7DjBOh8l7pfuaAKOrPIbo6Kdaa1f4Sl/view?usp=sharing

[MankaranSingh]

@PrithiveenKumaarRamkumar @SrinivasuluCharupally
I am not able to reproduce the issue. It would be helpful if you exactly mention what lines are causing segmenation fault in the script.

[MankaranSingh]

@SrinivasuluCharupally the issue is related to the carla-py package provided in this repo. you can try uninstalling that and do pip install carla==0.9.12 and rerun this.

[SrinivasuluCharupally]

@MankaranSingh - the segfault occurs when try to import the carla from bridge.py script

Steps:

#pipenv shell
# scons
# python tools/sim/bridge.py
Segmentation fault

===> this seg fault occur from the line import carla in the tools/sim/bridge.py

[SrinivasuluCharupally]

BTW - We tried the above steps with pip install carla==0.9.12, pip3 install carla==0.9.12 and pip install -e carla-py.
Result is same all the time, seg fault while importing carla into the bridge.py.
I am sure some thing is missing in the README.

[HiHeXi]

I also encountered the same problem. Have you solved the problem? If it is resolved, I would like to ask for advice on a solution

@MankaranSingh @RaghavPrabhakar66 Able to successfully launch flowpilot app now :) bridge.py still renders a Segmentation fault (core dumped) error
flowpilot_app_journey.mp4

I also encountered the same problem. Have you solved the problem? If it is resolved, I would like to ask for advice on a solution

[MankaranSingh]

You can try installing carla like this carla-simulator/carla#1466 (comment)

[SrinivasuluCharupally]

@MankaranSingh - could you confirm these steps are correct for each terminal. We are running every thing from the pipenv shell, check the below steps

Terminal#1

pipenv shell
scons
./CarlaUE4.sh -RenderOffScreen

Terminal#2

pipenv shell
launch_flowpilot.sh

Terminal#3

pipenv shell
python tool/sim/bridge.py

[MankaranSingh]

Your steps are correct. you just need a working carla-python api. Please check this out carla-simulator/carla#1466 (comment)

[HiHeXi]

你的步骤是正确的。 你只需要一个工作carla-蟒蛇。 请看看这个 卡拉-模拟器/卡拉#1466(注释)

This method won't work

[PrithiveenKumaarRamkumar]

@MankaranSingh
In carla-simulator/carla#1466 (comment) it is recommended to install carla by extracting the .egg files, but the .egg files are for python3.7.

However, activating the flowpilot environment pipenv shell takes you to python3.9 environment

The Carla build documentation https://carla.readthedocs.io/en/latest/build_linux/ and discussions carla-simulator/carla#4519 says that the PythonAPI is supported till python3.8. I am now able to import carla successfully in python3.7 and python3.8

Using python3.8 I was able to generate the build using scons and run the other scripts successfully but bridge.py renders the following traceback:

Traceback (most recent call last):
  File "tools/sim/bridge.py", line 28, in <module>
    pm = messaging.PubMaster(['roadCameraState', 'roadCameraBuffer', 'wideRoadCameraState',
  File "/home/che7/flowpilot/cereal/messaging/__init__.py", line 257, in __init__
    self.sock[s] = pub_sock(s)
  File "/home/che7/flowpilot/cereal/messaging/__init__.py", line 47, in pub_sock
    sock.connect(context, endpoint)
  File "cereal/messaging/messaging_pyx.pyx", line 139, in cereal.messaging.messaging_pyx.PubSocket.connect
    raise MultiplePublishersError
cereal.messaging.messaging_pyx.MultiplePublishersError

[MankaranSingh]

@PrithiveenKumaarRamkumar for MultiplePublishersError, you'd need to comment out the ROAD_CAMERA_SOURCE and wide camera source. This will not start the video publishers and allow publihers from carla bridge script to take over

[PrithiveenKumaarRamkumar]

@MankaranSingh Now we are able to run Carla, launch the flowpilot app and run bridge.py from the python3.8 environment but we are not able to "take control of the wheel" in the flowpilot app. Are we missing something here?
https://drive.google.com/file/d/1gLtdT1o6qeKijSyk_YwiPW62gROMnFJn/view?usp=sharing

[MankaranSingh]

https://github.com/flowdriveai/flowpilot/wiki/Running-On-Simulation

see the last step. The input keys would be taken in the bridge.py terminal

[SrinivasuluCharupally]

@MankaranSingh - We are able to control the Carla output with 1, 2 and W-A-D keys.
Could you tell us how to render with our own video with the environment, we have a local traffic video, would like to see the flowdriver output for that.
Let us know the details.

[SrinivasuluCharupally]

@MankaranSingh - could you guide us on this ?

[MankaranSingh]

The video can be specified in ROAD_CAMERA_SOURCE in launch script. By default, flowpilot desktop version run on a video only.

[SrinivasuluCharupally]

@MankaranSingh - do we have to launch all the three terminals with the "ROAD_CAMERA_SOURCE" enabled ?
The reason is, we noticed following error with ROAD_CAMERA_SOURCE enabled , you suggested to disable it /comment out last time.

Traceback (most recent call last):
File "tools/sim/bridge.py", line 28, in
pm = messaging.PubMaster(['roadCameraState', 'roadCameraBuffer', 'wideRoadCameraState',
File "/home/che7/flowpilot/cereal/messaging/init.py", line 257, in init
self.sock[s] = pub_sock(s)
File "/home/che7/flowpilot/cereal/messaging/init.py", line 47, in pub_sock
sock.connect(context, endpoint)
File "cereal/messaging/messaging_pyx.pyx", line 139, in cereal.messaging.messaging_pyx.PubSocket.connect
raise MultiplePublishersError
cereal.messaging.messaging_pyx.MultiplePublishersError

[MankaranSingh]

ROAD_CAMERA_SOURCE needs to be disabled to accept stream from carla simulation. do not run carla or or bridge.py if you want to run on video

…

On Sun, Oct 1, 2023 at 2:06 PM Srinivasulu Charupally < ***@***.***> wrote: @MankaranSingh <https://github.com/MankaranSingh> - do we have to launch all the three terminals with the "ROAD_CAMERA_SOURCE" enabled ? The reason is, we noticed following error with ROAD_CAMERA_SOURCE enabled , you suggested to disable it /comment out. Traceback (most recent call last): File "tools/sim/bridge.py", line 28, in pm = messaging.PubMaster(['roadCameraState', 'roadCameraBuffer', 'wideRoadCameraState', — Reply to this email directly, view it on GitHub <#38 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKHLNLGCXAN7HSIWLOSQWCLX5ETRHANCNFSM6AAAAAA2YQM3PE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[SrinivasuluCharupally]

Hi @MankaranSingh - Sorry for the late repose, we were busy with other projects.
Still we are not able to render the video.

script:

Attaching our "launch_flowpilot.sh" script.

steps:

pipenv shell
scons
./launch_flowpilot.sh

output

attaching screen shot

launch_flowpilot.py.txt

[PrithiveenKumaarRamkumar]

Hi @MankaranSingh
Do we have to make changes on these scripts for using user's input video?
https://github.com/flowdriveai/flowpilot/blob/master/selfdrive/loggerd/video_process.py
https://github.com/flowdriveai/flowpilot/blob/master/selfdrive/loggerd/config.py

[MankaranSingh]

The default launchfile when you download it freshly is taking in video input only. you do not need to run bridge.py.
after launch just run scripts/dummy_publishers.py in seperate terminal

[SrinivasuluCharupally]

Thanks @MankaranSingh
I am able to run it now. Attaching screen shots
one more question : Would it display the bounding boxes on detected objects ?

?

[MankaranSingh]

This is an end to end driving model. this dosent outputs bounding boxes

[SrinivasuluCharupally]

@MankaranSingh - I have downloaded a video from web, mp4 format, and tried to render the video under flowDrive, but seeing following error. You have any idea ?

Video download link : https://www.pexels.com/video/cars-traveling-on-expressway-5382495/

[MankaranSingh]

convert video to avi format.

[SrinivasuluCharupally]

@MankaranSingh - I tried but did not work.
Does it really requires Calibration & CAN signals ?
it shows "CalibrationError" & "canError".

[piyushmz]

Hey @MankaranSingh
I followed the steps in here https://github.com/flowdriveai/flowpilot/wiki/Running-On-Simulation for running on desktop. I get flowpilot in infinite loop.
ubuntu 20.04
nvidia-driver-535(proprietary tested)
carla= 0.9.12
I've given ROAD_CAMERA_SOURCE="test.avi" and Iaunched flowpilot following is the o/p I get on after running.

[main] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 2
[main] INFO org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory - Binary level Generic x86 optimization level AVX/AVX2
[main] INFO org.nd4j.nativeblas.Nd4jBlas - Number of threads used for OpenMP BLAS: 2
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Cores: [24]; Memory: [7.8GB];
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
[main] INFO org.nd4j.linalg.cpu.nativecpu.CpuBackend - Backend build information:
 GCC: "7.5.0"
STD version: 201103L
DEFAULT_ENGINE: samediff::ENGINE_CPU
HAVE_FLATBUFFERS
HAVE_OPENBLAS
[ERROR:0] global /home/runner/work/opencv/opencv/opencv-4.5.1/modules/videoio/src/cap.cpp (142) open VIDEOIO(CV_IMAGES): raised OpenCV exception:

OpenCV(4.5.1) /home/runner/work/opencv/opencv/opencv-4.5.1/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): /home/rp/Downloads/testhd.avi in function 'icvExtractPattern'


WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by ai.flow.common.BufferUtils (file:/home/rp/flowpilot/common/build/libs/common-0.0.1.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of ai.flow.common.BufferUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[main] INFO ai.flow.launcher.FlowInitd - Connected to the FlowInit daemon
[ERROR:0] global /home/runner/work/opencv/opencv/opencv-4.5.1/modules/videoio/src/cap.cpp (142) open VIDEOIO(CV_IMAGES): raised OpenCV exception:

OpenCV(4.5.1) /home/runner/work/opencv/opencv/opencv-4.5.1/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): /home/rp/flowpilot/selfdrive/assets/videos/intro.avi in function 'icvExtractPattern'


Exception in thread "camerad:roadCameraState" CvException [org.opencv.core.CvException: cv::Exception: OpenCV(4.5.1) /home/runner/work/opencv/opencv/opencv-4.5.1/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
]
        at org.opencv.imgproc.Imgproc.cvtColor_1(Native Method)
        at org.opencv.imgproc.Imgproc.cvtColor(Imgproc.java:6842)
        at ai.flow.sensor.camera.CameraManager.processFrame(CameraManager.java:112)
        at ai.flow.sensor.camera.CameraManager.run(CameraManager.java:165)
        at java.base/java.lang.Thread.run(Thread.java:829)

Is there any hardcoded path like /home/runner/* somewhere in flowpilot

If I comment out ROAD_CAMERA_SOURCE and start carla, flowpilot is always into infinite loop and nothing is detected.

[MankaranSingh]

@piyushmz what is test.avi ? where are you placing it ?

[piyushmz]

so I was testing with a downloaded mp4 file which I converted to avi and I placed the path in ROAD_CAMERA_SOURCE but it didn't work. The selfdrive/assets/tmp does work with flowpilot. As of bridging of carla with flowpilot I get segmentation fault.