Seems u* tools never work with namespaced processs

[keyolk]

Hi, trying to tracing python USDT wiith u*tools but stucked,
Chaning probe path is work with uprobe but seems it never work with USDT

$ uname -r
6.1.109

$ cat /etc/os-release
NAME=Bottlerocket
ID=bottlerocket
VERSION="1.25.0 (aws-k8s-1.30)"
PRETTY_NAME="Bottlerocket OS 1.25.0 (aws-k8s-1.30)"
VARIANT_ID=aws-k8s-1.30
VERSION_ID=1.25.0
BUILD_ID=388e1050
HOME_URL="https://github.com/bottlerocket-os/bottlerocket"
SUPPORT_URL="https://github.com/bottlerocket-os/bottlerocket/discussions"
BUG_REPORT_URL="https://github.com/bottlerocket-os/bottlerocket/issues"
DOCUMENTATION_URL="https://bottlerocket.dev"

$ tplist-bpfcc -p 146304
/proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0 python:function__entry
/proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0 python:function__return

updating the ucalls source

elif language == "python":
-    entry_probe = "function__entry"
+    entry_probe = "/proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0 python:function__entry"
-    return_probe = "function__return"
+    return_probe = "/proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0 python:function__return"

$ ./ucalls -l python -mLS 146304
Failed to enable USDT probe '/proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0 python:function__entry':
the specified pid might not contain the given language's runtime,
or the runtime was not built with the required USDT probes. Look
for a configure flag similar to --with-dtrace or --enable-dtrace.
To check which probes are present in the process, use the tplist tool.

Tried wth /proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0:python:function__entry" too which from bpftrace -l but no help

Any point to check? : (

[yonghong-song]

there are already some explanations. The python binary is not built with --with-dtrace or --enable-dtrace.

[keyolk]

@yonghong-song
The python I test built with dtrace already. No issue with bpftrace but got the error with ucalls

what I tested with bpftrace

bpftrace -e 'usdt:/proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0:python:function__entry { printf("%s %s\n", probe, str(arg0)); }'

it works without problem

[yonghong-song]

Maybe you can debug more by yourself from source?
The related failure code is in src/python/bcc/usdt.py

    def enable_probe(self, probe, fn_name):
        probe_parts = probe.split(":", 1)
        if len(probe_parts) == 1:
            ret = lib.bcc_usdt_enable_probe(
                self.context, probe.encode('ascii'), fn_name.encode('ascii'))
        else:
            (provider_name, probe_name) = probe_parts
            ret = lib.bcc_usdt_enable_fully_specified_probe(
                self.context, provider_name.encode('ascii'), probe_name.encode('ascii'),
                fn_name.encode('ascii'))

        if ret != 0:
            raise USDTException(
"""Failed to enable USDT probe '%s':
the specified pid might not contain the given language's runtime,
or the runtime was not built with the required USDT probes. Look
for a configure flag similar to --with-dtrace or --enable-dtrace.
To check which probes are present in the process, use the tplist tool.
""" % probe)

[keyolk]

from my attempt the provider_name of bcc_usdt_enable_fully_specified_probe is /proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0 python

and I also tried to make it to be /proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0 but no luck.

provider_name: /proc/146304/root/usr/lib/libpython3.8-pyston2.3.so.1.0
probe_name: function__entry

I guess I can check how the bpftrace does but the function bcc_usdt_enable_fully_specified_probe never been called in bpftrace

What I tried is do the below again

bpftrace -e 'usdt:/proc/329041/root/usr/lib/libpython3.8-pyston2.3.so.1.0:python:function__entry { printf("hello\n"); exit(); }'
WARNING: BPFTRACE_STRLEN is deprecated. Use BPFTRACE_MAX_STRLEN instead.
Attaching 1 probe...
hello

and check this with another bpftrace session

bpftrace -e 'uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_* { printf("%s %s %s\n",probe,str(arg1),str(arg2)); } '
WARNING: BPFTRACE_STRLEN is deprecated. Use BPFTRACE_MAX_STRLEN instead.
Attaching 14 probes...
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_new_frompid
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_foreach ?#
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_close
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_new_frompid
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_foreach ?#
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_close
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_new_frompid
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_foreach ?#
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_close
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_new_frompath P
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_foreach ?#
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_close
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_new_frompath probe_function__entry_1
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_get_location python function__entry
uprobe:/lib/aarch64-linux-gnu/libbcc_bpf.so.0:bcc_usdt_close

what the provider_name should be here?