Getting Error in installing vllm on Nvidia Jetson AGX ORIN

[MausamJain]

Getting Error in installing vllm on Nvidia Jetson AGX ORIN
I have tried installation using pip , source git source and building setup.py also but could not succeed.
Below are the details about my setup -

jtop 4.2.8 (c) 2024, Raffaello Bonghi [[email protected]]
Website: https://rnext.it/jetson_stats
Platform
Machine: aarch64
Serial Number: [s❘XX CLICK TO READ XXX] Hardware
Model: NVIDIA Jetson AGX Orin Developer Kit
Distribution: Ubuntu 22.04 Jammy Jellyfish 699-level Part Number: 699-13701-0005-500 S.0
Release: 5.15.136-tegra
System: Linux
Python: 3.10.12
Libraries
CUDA: 12.2.140
CUDNN: 8.9.4.25
TensorRT: 8.6.2.3
VPI: 3.1.5
Vulkan: 1.3.204
OpenCV: 4.8.0 with CUDA: NO
P-Number: p3701-0005
Module: NVIDIA Jetson AGX Orin (64GB ram) SoC: tegra234
CUDA Arch BIN: 8.7
L4T: 36.3.0 Jetpack: 6.0
Hostname: ubuntu Interfaces
eth0: 10.0.1.30 docker0: 172.17.0.1
Operating System: Linux - 5.15.136-tegra - aarch64
Compiler: /usr/bin/c++
Compiler Version: 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)

With pip -

(bot_venv) lftds@ubuntu:~/Documents/copilot$ pip install vllm
Collecting vllm
Using cached vllm-0.5.1.tar.gz (790 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [16 lines of output]
Traceback (most recent call last):
File "/home/lftds/Documents/copilot/bot_venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/home/lftds/Documents/copilot/bot_venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/lftds/Documents/copilot/bot_venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/tmp/pip-build-env-wsi_ghvs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
File "/tmp/pip-build-env-wsi_ghvs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-wsi_ghvs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 313, in run_setup
exec(code, locals())
File "", line 432, in
File "", line 353, in get_vllm_version
RuntimeError: Unknown runtime environment
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

From git source -

bot_venv) lftds@ubuntu:/Documents/copilot$ git clone https://github.com/vllm-project/vllm
Cloning into 'vllm'...
remote: Enumerating objects: 22588, done.
remote: Total 22588 (delta 0), reused 0 (delta 0), pack-reused 22588
Receiving objects: 100% (22588/22588), 21.67 MiB | 8.61 MiB/s, done.
Resolving deltas: 100% (16741/16741), done.
(bot_venv) lftds@ubuntu:/Documents/copilot$ cd vllm/
(bot_venv) lftds@ubuntu:~/Documents/copilot/vllm$ pip install .
Processing /home/lftds/Documents/copilot/vllm
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [16 lines of output]
Traceback (most recent call last):
File "/home/lftds/Documents/copilot/bot_venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/home/lftds/Documents/copilot/bot_venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/lftds/Documents/copilot/bot_venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/tmp/pip-build-env-x2sad19s/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
File "/tmp/pip-build-env-x2sad19s/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-x2sad19s/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 313, in run_setup
exec(code, locals())
File "", line 432, in
File "", line 353, in get_vllm_version
RuntimeError: Unknown runtime environment
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Building from source.py

(bot_venv) lftds@ubuntu:~/Documents/copilot/vllm$ python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-aarch64-cpython-310
creating build/lib.linux-aarch64-cpython-310/vllm
copying vllm/tracing.py -> build/lib.linux-aarch64-cpython-310/vllm
copying vllm/model_executor/layers/fused_moe/configs/E=8,N=7168,device_name=AMD_Instinct_MI300X.json -> build/lib.linux-aarch64-cpython-310/vllm/model_executor/layers/fused_moe/configs
copying vllm/model_executor/layers/fused_moe/configs/E=8,N=14336,device_name=AMD_Instinct_MI300X.json -> build/lib.linux-aarch64-cpython-310/vllm/model_executor/layers/fused_moe/configs
running build_ext
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type: RelWithDebInfo
-- Target device: cuda
-- Could NOT find Python (missing: Python_INCLUDE_DIRS Interpreter Development.Module Development.SABIModule) (found version "3.10.12")
CMake Error at cmake/utils.cmake:10 (message):
Unable to find python matching:
/home/lftds/Documents/copilot/bot_venv/bin/python.
Call Stack (most recent call first):
CMakeLists.txt:43 (find_python_from_executable)

-- Configuring incomplete, errors occurred!
See also "/home/lftds/Documents/copilot/vllm/build/temp.linux-aarch64-cpython-310/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
File "/home/lftds/Documents/copilot/vllm/setup.py", line 430, in
setup(
File "/home/lftds/Documents/copilot/bot_venv/lib/python3.10/site-packages/setuptools/init.py", line 103, in setup
return distutils.core.setup(**attrs) .
.
.
.
.
.
.
.
.
.
.
File "/home/lftds/Documents/copilot/vllm/setup.py", line 175, in configure
subprocess.check_call(
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/lftds/Documents/copilot/vllm', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/lftds/Documents/copilot/vllm/build/lib.linux-aarch64-cpython-310/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=build/temp.linux-aarch64-cpython-310', '-DVLLM_TARGET_DEVICE=cuda', '-DVLLM_PYTHON_EXECUTABLE=/home/lftds/Documents/copilot/bot_venv/bin/python', '-DNVCC_THREADS=1']' returned non-zero exit status 1.

(bot_venv) lftds@ubuntu:~/Documents/copilot/vllm$ cmake -DPYTHON_EXECUTABLE=/home/lftds/Documents/copilot/bot_venv/bin/python ..
CMake Error: The source directory "/home/lftds/Documents/copilot" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.

(bot_venv) lftds@ubuntu:~/Documents/copilot/vllm$ cd /home/lftds/Documents/copilot/vllm
mkdir -p build
cd build
cmake -DPYTHON_EXECUTABLE=/home/lftds/Documents/copilot/bot_venv/bin/python -DPYTHON_INCLUDE_DIR=/usr/include/python3.10 -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.10.so ..
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type:
-- Target device: cuda
CMake Error at CMakeLists.txt:45 (message):
Please set VLLM_PYTHON_EXECUTABLE to the path of the desired python version
before running cmake configure.

-- Configuring incomplete, errors occurred!
See also "/home/lftds/Documents/copilot/vllm/build/CMakeFiles/CMakeOutput.log".