Can't link in /librcl_interfaces__rosidl_generator_py.so due to it not linking in libpython #119

tony-p · 2023-12-19T17:04:36Z

Solution to issue cannot be found in the documentation.

I checked the documentation.

Issue

I'm having an issue on linux (humble) that when it tries to link in the librcl_interfaces__rosidl_generator_py.so it is missing a number of symbols resulting in undefined reference symbols

env/lib/librcl_interfaces__rosidl_generator_py.so: undefined reference to `PyLong_AsSize_t'

It is essentially the same problem as this https://robotics.stackexchange.com/questions/25039/generated-rosidl-generator-py-so-files-issue

And it is the same for the robostack provided version, however the workarounds don't seem to work and I think it is because it is already too late and should have been applied to these packages or the ones in between.

$ ldd -r .pixi/env/lib/librcl_interfaces__rosidl_generator_py.so
	linux-vdso.so.1 (0x00007ffcd3fd2000)
	libbuiltin_interfaces__rosidl_generator_py.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so (0x00007f51715d8000)
	librcl_interfaces__rosidl_generator_c.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./librcl_interfaces__rosidl_generator_c.so (0x00007f51715be000)
	librosidl_runtime_c.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./librosidl_runtime_c.so (0x00007f51715b2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5171200000)
	libbuiltin_interfaces__rosidl_generator_c.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/././libbuiltin_interfaces__rosidl_generator_c.so (0x00007f517159b000)
	librcutils.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/././librcutils.so (0x00007f5171584000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f51715ef000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f517157f000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f517157a000)
undefined symbol: PyObject_GetAttrString	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_SetAttrString	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_Dealloc	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromLong	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsLong	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyImport_ImportModule	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallObject	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLong	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLong	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyList_New	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLongLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBuffer_Release	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLongLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_GetAttrString	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyUnicode_AsUTF8String	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBytes_FromStringAndSize	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_SetAttrString	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBuffer_ToContiguous	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_Dealloc	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyExc_RuntimeError	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyErr_SetString	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyFloat_FromDouble	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PySequence_Size	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CheckBuffer	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBool_FromLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsSize_t	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyImport_ImportModule	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallObject	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyList_SetItem	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallFunctionObjArgs	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_Size	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyUnicode_DecodeUTF8	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_TrueStruct	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_GetBuffer	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsLongLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromLongLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PySequence_Fast	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)

Would patching the librcl_interfaces__rosidl_generator_py build to force link in libpython3 be the best solution here?

Installed packages

pixi Lock file is too long too paste here, will try find a more concise way to paste, but the relevant package is

- platform: linux-64
  name: ros-humble-rcl-interfaces
  version: 1.2.1
  category: main
  manager: conda
  dependencies:
  - libgcc-ng >=12
  - libgcc-ng >=12
  - libstdcxx-ng >=12
  - numpy >=1.21.6,<2.0a0
  - python
  - python_abi 3.10.* *_cp310
  - ros-humble-builtin-interfaces
  - ros-humble-ros-workspace
  - ros-humble-rosidl-default-runtime
  - ros2-distro-mutex 0.3.*
  - ros2-distro-mutex 0.3.* humble
  url: https://repo.prefix.dev/robostack-staging/linux-64/ros-humble-rcl-interfaces-1.2.1-py310h7c61026_3.tar.bz2
  hash:
    md5: 7123a8c0b2d39745f42ee72e444e8caf
    sha256: d66ad32b9a592a056b7b6a4eda2840a3b67d3b506a7ee3e29528bd81da9731c5
  build: py310h7c61026_3
  arch: x86_64
  subdir: linux-64
  build_number: 3
  size: 343519
  timestamp: 1675688249136

Environment info

pixi info
pixi 0.10.0

Platform            : linux-64
Virtual packages    : __unix=0=0
                    : __linux=6.2.0=0
                    : __glibc=2.35=0
                    : __archspec=1=x86_64
Cache dir           : /home/me/.cache/rattler/cache
Auth storage        : /home/me/.rattler/rattler_auth_store.json

Project
------------

Manifest file       : /home/me/workspace/project/pixi.toml
Dependency count    : 49
Last updated        : 19-12-2023 16:24:34
Target platforms    : win-64
                    : linux-64

The text was updated successfully, but these errors were encountered:

tony-p · 2024-01-03T08:59:39Z

Update: force linking in the output of $(python3-config --ldflags --embed) (from the linked issue) in the failing package does work, although as a temporary fix for another linking issue I had added the env/lib in LD_LIBRARY_PATH and then it tried to link in the wrong version of 'libfreebl3.so` which required a newer glibc

Tobias-Fischer · 2024-02-06T05:17:46Z

Hi @tony-p - could you please try again with the recently built updated packages?

tony-p · 2024-02-06T22:58:09Z

Tried again very naively and same issue occurs however checked and is taking build 3 instead of 5 of rcl-interfaces 1.2.1 so will look into a bit more when I have a little more time

tony-p · 2024-02-07T10:27:36Z

Dug a little deeper, and now have the build 5 and end up with the same errors. Further the ldd shows that these still aren't linked (below)

Further I encountered the following issues:

for some reason the dependency resolution was choosing an old version of sysroot that had libc v2.12. Forcing that sysroot >=2.17 solved it without other dependency conflicts so don't know why it wasn't chosen in the first place. This also means that there is the sysroot package and the virtual __glibc package both providing libc.
many ament_*lint executables are in the lib/<tool> directory which results in them not being found by cmake as they are not on the path. Manually adding each tool to the path (temporarily) solved that issue

ldd -r .pixi/env/lib/librcl_interfaces__rosidl_generator_py.so
        linux-vdso.so.1 (0x00007ffc8c0c7000)
        libbuiltin_interfaces__rosidl_generator_py.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so (0x00007f375c73b000)
        librcl_interfaces__rosidl_generator_c.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./librcl_interfaces__rosidl_generator_c.so (0x00007f375c721000)
        librosidl_runtime_c.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./librosidl_runtime_c.so (0x00007f375c715000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f375c4e6000)
        libbuiltin_interfaces__rosidl_generator_c.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/././libbuiltin_interfaces__rosidl_generator_c.so (0x00007f375c4de000)
        librcutils.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/././librcutils.so (0x00007f375c4c7000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f375c751000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f375c4c2000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f375c4bd000)
undefined symbol: PyObject_GetAttrString        (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_SetAttrString        (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_Dealloc   (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromLong       (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsLong (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyImport_ImportModule (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallObject   (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLong (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLong       (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyList_New    (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLongLong   (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBuffer_Release      (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLongLong     (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_GetAttrString        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyUnicode_AsUTF8String        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBytes_FromStringAndSize     (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_SetAttrString        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBuffer_ToContiguous (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_Dealloc   (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyExc_RuntimeError    (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyErr_SetString       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyFloat_FromDouble    (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PySequence_Size       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CheckBuffer  (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBool_FromLong       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsSize_t       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyImport_ImportModule (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallObject   (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyList_SetItem        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallFunctionObjArgs  (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_Size (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyUnicode_DecodeUTF8  (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_TrueStruct        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_GetBuffer    (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLong (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLong       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsLongLong     (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromLongLong   (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PySequence_Fast       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)

Tobias-Fischer · 2024-02-09T08:58:01Z

I wonder if this should be public:

ros-humble/patch/ros-humble-rosidl-generator-py.patch

Line 61 in 52756fd

    
           +target_link_libraries(${_target_name_lib} PRIVATE Python3::NumPy Python3::Module)

Any ideas @traversaro @wolfv?

Tobias-Fischer · 2024-02-28T06:18:27Z

I've just pushed 7d0d8be which I have locally confirmed fixes this issue. If you rebuild ros-humble-rosidl-generator-py and then ros-humble-rcl-interfaces the issue will go away. Unfortunately at the moment I have not the capacity to build this in CI; but it would be easy to do (please see Contributing guidelines if you'd like to open a PR @tony-p).

tony-p · 2024-02-28T20:49:06Z

Great thanks, will try find some time to try soon (also limited capacity) and if fixes and have time will definitely look into the CI build

tony-p added the bug Something isn't working label Dec 19, 2023

Tobias-Fischer added the linux-64 label Feb 6, 2024

Tobias-Fischer added the needs-rebuild Issue has been fixed but packages require rebuild label Feb 29, 2024

tony-p mentioned this issue Apr 5, 2024

add additional patches for gcc 13 and fix rosidl py generator patch on linux #169

Merged

tony-p mentioned this issue Oct 8, 2024

Building with WSL does not work BehaviorTree/BehaviorTree.CPP#868

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't link in /librcl_interfaces__rosidl_generator_py.so due to it not linking in libpython #119

Can't link in /librcl_interfaces__rosidl_generator_py.so due to it not linking in libpython #119

tony-p commented Dec 19, 2023

tony-p commented Jan 3, 2024

Tobias-Fischer commented Feb 6, 2024

tony-p commented Feb 6, 2024

tony-p commented Feb 7, 2024 •

edited

Loading

Tobias-Fischer commented Feb 9, 2024

Tobias-Fischer commented Feb 28, 2024

tony-p commented Feb 28, 2024

Can't link in /librcl_interfaces__rosidl_generator_py.so due to it not linking in libpython #119

Can't link in /librcl_interfaces__rosidl_generator_py.so due to it not linking in libpython #119

Comments

tony-p commented Dec 19, 2023

Solution to issue cannot be found in the documentation.

Issue

Installed packages

Environment info

tony-p commented Jan 3, 2024

Tobias-Fischer commented Feb 6, 2024

tony-p commented Feb 6, 2024

tony-p commented Feb 7, 2024 • edited Loading

Tobias-Fischer commented Feb 9, 2024

Tobias-Fischer commented Feb 28, 2024

tony-p commented Feb 28, 2024

tony-p commented Feb 7, 2024 •

edited

Loading