DWARFless Debugging

[brenns10]

Last updated: 2024-10-11

This issue tracks support for non-DWARF sources of debugging information: specifically for the Linux kernel, but hopefully including userspace as we go. I'm editing this initial issue comment as the project takes shape, so hopefully this can provide at-a-glance status information.

Overview

This issue tracks the use of non-DWARF formats for debuginfo. Namely: CTF for types, ORC (without a debuginfo file) for stack unwinding, BTF for types as well, and kallsyms for symbols.

Objectives

The currently agreed upon end goal here is to get a pluggable symbol finder and vmlinux kallsyms implementation merged and available by default for Drgn. These are useful in and of themselves:

• Even without type info, a vmlinux kallsyms implementation would allow people to lookup symbols, a major improvement over the current state of things (in which only Program.read* functionality would be available).
• It's relatively common to have vmlinux type info, but not module info. A pluggable system would allow a module symbol implementation that uses the module kallsyms and module exported symbols.

Once these are available, we will get a basic CTF implementation for Linux kernel merged. The basic ground rules here are that it will be disabled at compile-time in the PyPI wheel distributions, and won't muddle into the internals of Drgn. Essentially, there should be some build-system related changes, and a file named ctf.c, and maybe some python wrappers, and that's it.

Here are some non-goals at the moment. They may be revisited in time.

1. CTF support for userspace programs. Any program compiled with -gctf has a .ctf section. Compare that to the kernel implementations, which now just create a vmlinux.ctfa (ctfa = CTF Archive) file. As of now, the CTF implementation does support very simple userspace cores in order to add simple unit tests. However, proper support will need better integration with the drgn debug info system, so it won't be officially supported for the initial step.
2. Automatic detection and use of CTF data. This would require tangling the CTF implementation into the core a bit more than we want to do initially. Also, it might conflict with some of the work on the Module API.
3. Enabling CTF support in PyPI wheels.

Roadmap

1. First pull request: Add VMCOREINFO to Linux special objects #316 Add VMCOREINFO to the Linux special objects. This is not controversial, it's just a nice piece of information to expose to Python helpers. The current design of my helpers does need this, but even if it didn't, it would be a useful piece.
2. Second pull request: Pluggable Symbol finder API, with Python support #241 Pluggable Symbol finder API. Allows C and Python to register "symbol finders". In review
3. Third pull request: Add SymbolIndex and kallsyms helpers #388 Kallsyms symbol finders.
4. CTF type & object finder. (see ctf branch).
5. Support ORC unwinding without ELF files. (see ctf branch).
6. BTF type & object finder.

Current branches

These are links to branches that contain my current work, and they do roughly correspond to different points in the roadmap above. They are stacked, each one building on the prior one. They are subject to being rebased and force pushed at any time.

1. symbol_finder - this branch adds the pluggable symbol finder API.
2. kallsyms_finder - this branch adds the kallsyms implementation
3. ctf - this branch adds the CTF implementation. It also has the necessary plumbing to use ORC for unwinding, without needing to read it from ELF debuginfo files.
4. btf_2024 - this branch adds a small BTF implementation. Support for global variables (which I recently added to BTF via dwarves) is in progress, see the code for details.

If you take the latest branch (ctf) on an Oracle Linux 9 machine using UEK, then you should be able to build it and install it against a local kernel without installing any debuginfo packages!

sudo dnf config-manager --enable ol9_developer_EPEL
sudo dnf config-manager --enable ol9_developer
sudo dnf config-manager --enable ol9_appstream
sudo dnf config-manager --enable ol9_codeready_builder

sudo dnf install -y make autoconf automake libtool gcc-c++ git \
                    python3-devel elfutils-devel binutils-devel \
                    bzip2-devel zlib-devel xz-devel \
                    libkdumpfile-devel

cd drgn
python setup.py build_ext -i
sudo python -m drgn
# tada

Future Work

• BTF: I have a recently updated branch with some minimal BTF support. It could use some improvement. I'm also considering using libbpf in the future, rather than hand-coding the format support.
• Userspace CTF support needs some improvement. It exists in the current CTF branch in a very limited way (which helps in unit testing).

Old Branches & Work

I have created a few prototype branches on older drgn versions. Only the ones mentioned above are actively maintained and developed. The ones below are older and no longer maintained. For the most part, the commits in these branches were used as the basis for more recent branches, so it's not like the work is lost. The below list is from oldest to newest.

• btf_debuginfo - this was my original attempt at alternative debuginfo formats. It didn't have any symbol table support. I submitted Add BTF type finder #162 for this, and closed it as I worked on newer code.
• kallsyms_plus_btf - builds on the above with kallsyms support
• kallsyms_vmlinux - this contains the kallsyms implementation, without any BTF code. This went into its own pull request (Implement kallsyms for vmlinux only #177) which I am about to close as well since the branch is outdated.
• kallsyms_ctf - this is the old kallsyms implementation, with a preliminary, very hacky CTF implementation.

[brenns10]

Update today:

• Rewrote the first comment to give background and purpose, and then describe current branches, status, and prior branches.
• Closed Implement kallsyms for vmlinux only #177 and Add kernel module helpers #229, they are both contained in the branches here, and all of this is in a draft state until the module API comes in.

[brenns10]

Updated today:

• All the branches are rebased onto current main. I've been regularly force-pushing and rebasing them to keep them up-to-date, and to keep the commits related to each branch together. But not always commenting when I do so.

Some of the new things in there include:

• Using /proc/kallsyms as a fast-path for live kernels. This is an obvious optimization but it wasn't my top priority initially. It means that you don't need any of the new vmcoreinfo commits which were added in 6.0 in order to use it. (Of course, if you want to debug a vmcore, you'll need those commits).
• Added some more flexibility to the CTF linking. There's a lot of ways that libctf can be packaged, as static or dynamic libraries. Currently the flexibility is something like a "passthrough" of some linker flags provided to the configure script. It's not ideal though.
• Updated the wheel scripts to support libctf.
• Kallsyms now supports some newer changes for Rust's "long names"

[brenns10]

Updated today:

A tentative roadmap based on discussions with Omar. Link to the first (small) pull request of several in the series.

[brenns10]

Some more technical notes from the discussion so I remember them when I work on the relevant parts:

1. Ownership of symbol names is tricky once you start allowing Python code to create symbols -- or any code with shorter lifetimes than the Program. My current solution is to add a flag (name_owned) to the symbol object, which indicates that ownership of the string is passed into the symbol, and it will be freed when the symbol is destroyed. However, we discussed an alternative, where the Python layer creates a set of interned strings, and each time a symbol is created, we check if the string exists already. If so, we just use the interned copy. If not, we add it to the set and use it. After a bit of a back and forth, we couldn't really decide on which approach is better. I offered to implement the alternative where we intern the strings so we have a point of comparison.
2. The symbol finder API currently does not expose module information at all, but that would be extremely useful (or just necessary). When looking up symbols by address, we should unconditionally include the module information if not already present, since it should be a relatively cheap tree lookup. We can allow the symbol finder to take the module as an additional constraint. However, until the module API is exposed to Python, this will have to be omitted for the Python version of this API.
3. We were frequently nerd sniped in our discussion, and one mostly unrelated project / idea that I mentioned was to allow the user to override the vmcoreinfo (or the information contained therein) used by drgn. This would allow Drgn to more gracefully handle cases where the info is wrong or missing. I'm interested in this problem but it's fully unrelated to the task at hand :)

[brenns10]

Note to self: update CTF branch for 30ecdd9

[brenns10]

The ctf, symbol_finder, and kallsyms_finder branch are updated based on my latest rebase, which is on main, which is currently 3 commits ahead of 0.0.25, so this is more or less the 0.0.25 update.

• No changes for symbol_finder, it's still under review so everything is in PR Pluggable Symbol finder API, with Python support #241
• Changes for kallsyms_finder:
  • Name lookups are (finally) performed with a hash table, not a stupid linear search. I don't know why I waited so long to do this.
  • I squashed down the history a bit, because it wasn't terribly useful, and amending the history with the above change was difficult.
• Changes for ctf
  • Introduce our own hash table for mapping names to CTF (dict, id) pairs. Improves performance, simplifies some code, and has some nice correctness benefits.
  • Introduce a hash table mapping (dict, id) to a struct drgn_type * (I don't think there's any need to cache qualified_type since they're so simple). This is another great performance win.

With these changes, on one benchmark I have (which reads 100k elements from the dentry hash table), the CTF implementation now outperforms the DWARF:

• Running with DWARF debuginfo: 8.38s, using a maximum of 588.8 MiB of memory.
• Running with CTF debuginfo: 7.20s, using a maximum of 379.6 MiB of memory.

[brenns10]

I've updated the ctf branch with some new work that enables the use of ORC to unwind stacks when there are no ELF debuginfo files loaded.

As it is now, drgn reads ORC only from the ELF files it has already located. If there are no ELF debuginfo files, there's no DWARF. A consequence of this is that the unwinder is tightly coupled with the drgn_module API, and as it is now, when there is no ELF debuginfo file, there is no corresponding drgn_module with which to associate ORC info.

So my changes add a binary tree mapping address ranges directly to ORC information, and this can be used regardless of what debuginfo is loaded. I'd expect that in the future as this goes upstream, there will be a better solution (e.g. the module API refactor) which eliminate the need for this.

[brenns10]

Updated with the latest progress, #241 is merged and #388 is filed. Also updated some of the discussion related to ORC and userspace support (ORC support is now present in the ctf branch and userspace is partially supported in order to help with unit testing).

[oshaked1]

Hi @brenns10, first of all I want to say I really appreciate your work on this set of features. I am working on a project that will use drgn as an automatic incident response tool on production servers and these features will be very useful for me. Specifically, I'm very interested in BTF support as most servers I will be encountering do not come with CTF.

Can you give a quick update on the status of BTF support? Will it be available anytime soon? Is there any way I can help with it?

Thanks a lot!

[brenns10]

Hi @oshaked1 thanks, I'm glad this feature is interesting to you!

In terms of drgn development progress, BTF and CTF are both type formats, and so both of them require using kallsyms as a symbol table. Basically, steps 1-3 (as well as 5) on my roadmap are shared for both CTF and BTF -- the only real difference is which type format gets used. So once step 3 is done (#388), a usable BTF implementation could be written & merged independently of CTF. My very old BTF branch here will be a good place to start.

The major snag is that currently, the Linux kernel does not include BTF for kernel variables, just functions. So, while the BTF implementation could be updated and prepared for submission, it would not be useful until we get the Linux kernel (and its BTF generation program, pahole) to generate BTF for variables as well as functions. On the other hand CTF (which is not upstream yet, so it's another major snag) has that information already.

For my development plans, I'm prioritizing the CTF implementation first, but I do plan to work on BTF after that. If you or somebody else wanted to take up that work in order to see it merged faster, I'd be happy to help with design & review.

[oshaked1]

Hi @brenns10, thanks for the reply and sorry for the delay. For my use case, I can live without having types for kernel variables, as long as I can specify them manually. Would it be useful for the project if I were to contribute a solution that doesn't address the variable type issue? It would still be quite useful for use cases where the types of the variables being accessed are known, e.g. scripts. I could also leave my branch public so it can be used or built upon when the Linux kernel tooling is ready.

Regarding implementation - I saw you mentioned switching to using libbtf. Did you mean libbpf? Also, how up to date is the code in your BTF branch as far as integration with the drgn framework goes? Have there been significant changes to the type finder system?

[brenns10]

Would it be useful for the project if I were to contribute a solution that doesn't address the variable type issue?

The BTF type finder itself in drgn would look pretty much the same, regardless of whether Linux actually contained the types for the variables or not. So from that perspective, it would be useful! But without the Linux support, the code in drgn would not be terribly useful. Even though BTF currently contains many types, it only contains those referenced by the functions and percpu variables it covers. While this is a lot of types, it's not all types, so you wouldn't always be able to write scripts even if you specified the type for a variable.

Regardless, it would be helpful to have the BTF branch updated since we would love to be able to use BTF someday, and it's a step in the right direction.

Did you mean libbpf?

Yes, I meant libbpf, sorry! I saw that there are some BTF-related functions in it, and I haven't evaluated whether it is possible to use it for BPF.

Honestly, I think I could get the BTF branch up and running on a much more recent version of drgn over the course of a couple hours. There's not a ton of drgn-specific stuff that has changed; instead, it's mainly that my kallsyms support has improved a lot. It perhaps a bit silly of me to suggest you update it, given that I have all the context, and I haven't done much in terms of documenting things! I will try to take the time to take a crack at it tomorrow, and report back on whether it was as straightforward as I hope it is.

[brenns10]

Ok, I pushed a new branch btf_2024 which contains my updated BTF implementation. The work wasn't too difficult, the time spent on it was:

1. Rebasing the CTF branch on the latest main branch. This was necessary due to the new changes to the symbol finder API.
2. Updating the BTF implementation with a few minor API differences. The only major API difference was that type finders now accept a bitfield of requested type kinds, rather than being called once for each requested type kind.
3. Updated the BTF implementation to contain support for the new BTF_KIND_ENUM64 for 64-bit enums. To be fair this is entirely untested, but it "should work".

It's based directly on top of the CTF branch, so it can make use of the existing support for ORC & kallsyms. I'd recommend reading the commit messages of the two BTF-related commits at the top of this branch.

I tested it on a 5.15-based vmcore I had laying around, but you can also run it on the latest vmtest kernels. Here's an example:

python -m vmtest -k 6.9*
# within the virtual machine:
python -m drgn --no-default-symbols
>>> load_btf(prog)
>>>

Like I said, there's no general mapping from variable name to type, which means that for most variables, you'd need to explicitly specify the type. I added a var function as a syntax sugar for it.

>>> prog["slab_caches"]
Traceback (most recent call last):
  File "<console>", line 1, in <module>
KeyError: 'slab_caches'
>>> Object(prog, "struct list_head", address=prog.symbol("slab_caches").addres)(struct list_head){
        .next = (struct list_head *)0xffffa273022b8168,
        .prev = (struct list_head *)0xffffa27301042068,
}
>>> from drgn.helpers.linux.btf import var
>>> var(prog, "slab_caches", "struct list_head")
(struct list_head){
        .next = (struct list_head *)0xffffa273022b8168,
        .prev = (struct list_head *)0xffffa27301042068,
}

I did add a few core variable types into a special "hardcoded" object finder, because there are some kernel variables that drgn tries to access internally, and having the types handy prevents them from failing. But there's not really any guarantee that all the types you'd like to refer to will be present, unfortunately. However, it's enough to get drgn's built-in thread API to work, and thanks to the already present ORC plumbing, the stack tracing even works!

>>> for thread in prog.threads():
...     print(thread.object.comm.string_().decode())
...     print(thread.stack_trace())
...
init
#0  __schedule+0x4d0/0x512
#1  schedule+0x2a/0x41
#2  do_wait+0xcb/0xf5
#3  kernel_wait4+0xd8/0x131
#4  __do_sys_wait4+0x49/0x9e
#5  do_syscall_64+0x82/0xe0
#6  entry_SYSCALL_64_after_hwframe+0x76/0x7e
#7  0x7ff936820a7a
kthreadd
#0  __schedule+0x4d0/0x512
#1  schedule+0x2a/0x41
#2  kthreadd+0x72/0x11f
#3  ret_from_fork+0x20/0x35
#4  ret_from_fork_asm+0x1a/0x30
pool_workqueue_
#0  __schedule+0x4d0/0x512
#1  schedule+0x2a/0x41
#2  kthread_worker_fn+0x154/0x1b8
#3  kthread+0xe0/0xeb
#4  ret_from_fork+0x20/0x35
#5  ret_from_fork_asm+0x1a/0x30
kworker/R-rcu_g
#0  __schedule+0x4d0/0x512
#1  schedule+0x2a/0x41
#2  rescuer_thread+0x224/0x23e
#3  kthread+0xe0/0xeb
#4  ret_from_fork+0x20/0x35
#5  ret_from_fork_asm+0x1a/0x30
...

[oshaked1]

Looks great! Thanks so much for your effort, definitely would have taken me much longer to figure this out. The var function is just what I was looking for. I'm having some trouble building your branch, I'm using scripts/build_dists.sh, but after all wheels are built I get the following error:

Processing /tmp/manylinux_wheels/drgn-0.0.26+unknown-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Installing collected packages: drgn
Successfully installed drgn-0.0.26+unknown
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ /opt/python/cp310-cp310/bin/drgn --version
Traceback (most recent call last):
  File "/opt/python/cp310-cp310/bin/drgn", line 5, in <module>
    from drgn.cli import _main
  File "/opt/_internal/cpython-3.10.14/lib/python3.10/site-packages/drgn/__init__.py", line 49, in <module>
    from _drgn import (
ImportError: /opt/_internal/cpython-3.10.14/lib/python3.10/site-packages/_drgn.cpython-310-x86_64-linux-gnu.so: undefined symbol: ctf_dict_open

I get the same error when using the built wheel outside the build container as well.

Before that I also got the following error:

../../libdrgn/btf.c: In function ‘drgn_btf_lookup_enumval’:
../../libdrgn/btf.c:822:2: error: a label can only be part of a statement and a declaration is not a statement
  822 |  struct drgn_qualified_type qt;
      |  ^~~~~~
../../libdrgn/btf.c:823:2: error: expected expression before ‘struct’
  823 |  struct drgn_error *err = drgn_btf_type_create(bf, tid, &qt);
      |  ^~~~~~
../../libdrgn/btf.c:824:6: error: ‘err’ undeclared (first use in this function)
  824 |  if (err)
      |      ^~~
../../libdrgn/btf.c:824:6: note: each undeclared identifier is reported only once for each function it appears in

Which I fixed by moving the variable declarations to the beginning of the function.

[brenns10]

You can fix the undefined ctf_dict_open() issue by reverting commit 9639779 so that the build doesn't try to use CTF in the container. I haven't actually tried to do that in a while and I guess I need to revisit and debug that. You can also set CONFIGURE_FLAGS=--with-libctf=no for local builds.

Unfortunately libctf doesn't have pkg-config scripts and so every distro / system has slightly different linker flags, it's a bit difficult to get it work generally. Better to turn it off if you're just using BTF.

[brenns10]

Force-pushed btf_2024 with fixes:

• Moved the declarations to the top of the function -- it's strange that my compiler had no issues with that syntax.
• I've dropped the commit which adds libctf to the wheel building process. I wasn't intending to include it in the final drgn+CTF pull request, and it was untested on more recent versions of drgn.
• I've updated the autotools check for libctf to search for a symbol added in binutils 2.36, which is the minimum supported version. This prevents some other build errors.
• Finally, I've been working on ensuring the wheel build works when CTF is disabled. Unfortunately there are a few tests failures remaining (due to my added CTF test data files not being present in the source tarball). I'll see how to handle those, but for now builds work fine outside of the docker container. You can delete those tests or comment out the test run in the docker build if you want to work through it.

edit: and now I've fixed the test failures

[oshaked1]

It works now, thanks!

[brenns10]

Excellent! Your feedback has already been helpful, but I'd appreciate any more you have as you use it :)

[oshaked1]

Hi just reporting a tiny issue - the load_btf() function fails when there are no modules (encountered this on WSL where there are no kernel modules):

File "/usr/local/lib/python3.10/dist-packages/drgn/helpers/linux/btf.py", line 79, in load_btf
    module_finder = load_module_kallsyms(prog)
  File "/usr/local/lib/python3.10/dist-packages/drgn/helpers/linux/kallsyms.py", line 254, in load_module_kallsyms
    return SymbolIndex(all_symbols)
ValueError: symbol finder must contain at least one symbol

Everything works fine when ignoring the exception.

[brenns10]

Great timing, you're in luck. That restriction is being removed in the upstreaming, #388. I fixed it Friday and it will be in the next version of PR. I will rebase the CTF and BTF patch sets with those updates for the new revision as well.

[oshaked1]

Awesome, I finally got around to integrating your BTF patch into my project so I may have some additional feedback in the upcoming days.

I found myself adding quite a few hardcoded types that are required by existing helpers so I will share them with you when I'm done.

[brenns10]

That's great, I'm so glad you're finding this useful!

I'll be glad to incorporate any of the hard-coded types into the BTF branch. I'm still aiming to get the variable types included in the kernel BTF but improving the usability as it is now is a great temporary measure.

[oshaked1]

Here are the hardcoded types I added that are needed for some existing helpers, with some logic for types that depend on specific configurations:

drgn.helpers.linux.btf.HARDCODED_TYPES["slab_kset"] = "struct kset *"
drgn.helpers.linux.btf.HARDCODED_TYPES["slab_caches"] = "struct list_head"
drgn.helpers.linux.btf.HARDCODED_TYPES["min_low_pfn"] = "unsigned long"
drgn.helpers.linux.btf.HARDCODED_TYPES["max_pfn"] = "unsigned long"
drgn.helpers.linux.btf.HARDCODED_TYPES["saved_command_line"] = "char *"
drgn.helpers.linux.btf.HARDCODED_TYPES["net_namespace_list"] = "struct list_head"

# We have CONFIG_SPARSEMEM_EXTREME, mem_section type is `struct mem_section **`
try:
    self.prog.function('sparse_index_alloc')
    drgn.helpers.linux.btf.HARDCODED_TYPES["mem_section"] = "struct mem_section **"
# We don't have CONFIG_SPARSEMEM_EXTREME, mem_section type is `struct mem_section[NR_SECTION_ROOTS][SECTIONS_PER_ROOT]`.
# We ignore the sizes as they are calculated in `linux_kernel_get_vmemmap_address`.
# TODO: make sure this works, this isn't tested.
except LookupError:
    drgn.helpers.linux.btf.HARDCODED_TYPES["mem_section"] = "struct mem_section[0][0]"

# Kernel >= 6.9, vmap_nodes is used
try:
    self.prog.symbol("vmap_nodes")
    drgn.helpers.linux.btf.HARDCODED_TYPES["vmap_nodes"] = "struct vmap_node *"
# Kernel < 6.9, vmap_area_list is used
except LookupError:
    drgn.helpers.linux.btf.HARDCODED_TYPES["vmap_area_list"] = "struct list_head"

Incorporating variable types in the kernel BTF is a great idea, but it would still be valuable to have the essential hardcoded types in drgn for backward compatibility.

[brenns10]

I will say that the hardcoded types may have some issues when we finally go to review & merge the BTF portion of the branch. Not sure that they will stand up to code review!

Worst case I'm sure we could put them into the contrib directory, but I just wanted to let you know that, while I'm happy to add these into the branch, they may not make the final cut.

[oshaked1]

If it's a matter of just calling a function from the contrib directory then that's completely fine :)

[brenns10]

Sent a new patch series upstream for dwarves/pahole to add global variables to BTF generation.

This does interestingly expose a gap in the BTF support I've implemented so far: BTF contains a DATASEC type entry, which annotates each VAR with the data section it appears in, as well as the offset of the variable from that section's start. So far, I've been using the kallsyms symbol table to find addresses, and then looking up the VAR by name in the BTF, and using that to find variables. In my next iteration of the BTF patches for drgn I'll include native support for the DATASEC so that we don't necessarily have to rely on kallsyms.

[brenns10]

My patches enabling global variables have landed in the next branch of dwarves. There was one uncaught bug but we're getting that cleared up. If all goes well, there will be a future release of pahole soon where we can have global variables in the BTF, and then it would be a 1-line patch to add support for generating BTF of all global variables in the Linux kernel.

With that, I've updated the btf_2024 branch. I built Linux 6.11 with BTF +global_var enabled, and then generated a vmcore. With the btf_2024, no external piece of debuginfo (or hardcoded variable types) are required to run many of the contrib scripts. For example:

$ python -m drgn -c ~/repos/linux-upstream/vmcore.btf_global_vars --btf contrib/slabinfo.py
struct kmem_cache *  |         name         | active_objs  |   num_objs   | objsize  | objperslab  |  pageperslab
-------------------- | -------------------- | ------------ | ------------ | -------- | ----------- | -------------
0xffff9e40c211de00   | p9_req_t             |            0 |            0 |      160 |          25 |             1
0xffff9e40c211dd00   | kcopyd_job           |            0 |            0 |     3240 |          10 |             8
0xffff9e40c211dc00   | bio-120              |           32 |           32 |      128 |          32 |             1
0xffff9e40c211db00   | i915_vma_resource    |            0 |            0 |      320 |          25 |             2
0xffff9e40c211da00   | i915_dependency      |            0 |            0 |      128 |          32 |             1
....

So this is a pretty cool milestone!