Implement kallsyms for vmlinux only

[brenns10]

Hi Omar,

I've been reworking my very work-in-progress branch for BTF & kallsyms, and I sketched out my approach in #176. This pull request implements step 1 - kallsyms parsing for vmlinux only. Using symbols from vmcoreinfo (patches for that look to be accepted), this parses and reads the compressed kallsyms data and extracts it to some easily-queried arrays.

To make this patch have a functional, testable result, I've refactored the ELF symbol code which I had earlier worked on -- it is now modular. Then, I implemented the new symbol API for kallsyms. I'm not absolutely wedded to the particular API design I chose, but I wanted to start somewhere. I don't have a ton of work building off of this patch series yet, so we can iterate on the API if you have better ideas.

So now, we're able to load a vmcore without any debuginfo or ELF symbol table, and do the following:

$ venv/bin/drgn -c test3/vmcore
drgn 0.0.19+6.gf753cf7 (using Python 3.9.5, elfutils 0.180, with libkdumpfile)
Loaded symbols from kallsyms
For help, type help(drgn).
>>> import drgn
>>> from drgn import NULL, Object, cast, container_of, execscript, offsetof, reinterpret, sizeof
>>> from drgn.helpers.linux import *
>>> len(prog.symbols())
107551
>>> prog.symbol(0xffffffffb96203f0 + 20)
Symbol(name='cstate_pmu_event_init', address=0xffffffffb96203f0, size=0x180, binding=<SymbolBinding.UNKNOWN: 0>, kind=<SymbolKind.FUNC: 2>)

[brenns10]

Hm reminder for myself, this needs to go into a header. Sorry for missing that.

[brenns10]

Over the weekend Andrew Morton moved my vmcoreinfo updates into his mm-nonmm-stable queue, which I think will be included in his pull requests for the 5.20 merge window. So hopefully this will be applicable to upstream kernels soon!

[osandov]

Awesome! I'm currently working on adding some BPF tests to unblock #152, and then this will be the next thing I turn my focus to.

[osandov]

I gave this a high-level look and it looks really nice overall. The kallsyms format is more complicated than I expected, but there's not much we can do about that :)

The big thing we need to figure out is when we should load and use kallsyms. It looks like this PR loads the main vmlinux kallsyms if debug info for any module is missing, which makes sense for this proof of concept. But presumably, kallsyms is only useful if we don't have the symbols available for vmlinux, right? And similarly for kernel modules, we probably only want the kallsyms for modules whose symbols we didn't find? I'm thinking that the ideal strategy would be to lazily load kallsyms the first time you try to get look up a symbol in a module (vmlinux or kernel module) that doesn't have symbols available from a file. Otherwise, we're potentially going to be wasting work and memory loading the kallsyms data. What do you think? How expensive is loading kallsyms, CPU- and memory-wise?

If this is the strategy we want to go with, the mechanism to make it happen might be a little tricky. That'll need more discussion.

[brenns10]

Yeah, the integration with the other debuginfo sources is going to be the trickier part of this whole thing. I had envisioned this system as a binary choice - either all DWARF debuginfo with the ELF symbol table, or kallsyms with the BTF type info. I hadn't really given any thought to the option of using one to supplement the other when it's unavailable.

In terms of performance overhead, did a time bash -c 'for i in {1..200}; do drgn -c vmcore -s do_nothing.py; done' on both this branch and the commit this branch is based on:

This branch:
real    0m11.218s
user    0m9.710s
sys     0m1.507s

Base:
real    0m6.384s
user    0m5.303s
sys     0m1.078s

This gives an overhead of 24 milliseconds ((11.218 - 6.384) / 200 = 0.02417) for the kallsyms loading as currently written. In terms of memory, I loaded the same vmcore (it's 5.4 based, Oracle UEK6, but I think it's reasonably representative for an average distribution kernel) and saw the allocations were:

Symbol buffer length: 2546897
Symbol pointer array: 860648
Symbol type array: 107581

All told, about 3.35 MiB of allocations on this kernel, and of course this without modules. So there's real overhead here in CPU and memory, but it's not outrageous. The module symbol loading will add more memory and and a bit of CPU overhead, but should be easier on both, because there are fewer symbols and also none of the compression to undo.

I don't know where this leaves us regarding lazy loading kallsyms. Compared to DWARF handling, I'd assume this is still faster, so it makes sense to fallback to it when there's no DWARF. Module kallsyms could be supported without needing to do any of the complex parsing, so it should be faster, and it wouldn't depend on vmlinux kallsyms. However, module kallsyms would still depend on type information provided by DWARF or BTF. So we'd need to be mindful of all kinds of dependencies for using kallsyms, which might get a bit ... unwieldy.

I do think it could be valuable to put some of the policy into separate drgn.Program APIs that get driven by the Python wrapper. That gives people the flexibility to make their own wrapper with their own debuginfo policy. Rather than having load_debug_info() do magic and implement a policy that's set in stone. But I think that's probably a separate issue.

[brenns10]

To try to clarify dependencies a bit:

• vmlinux kallsyms: can be supported when (a) the vmcoreinfo note contains my extra symbols, or (b) we have symbol info already, which pretty much defeats the point.
• vmlinux BTF: needs to have some symbol information to find the BTF. In my vision, this comes from kallsyms, but I suppose it could come from an ELF file which contains no type information, but still has the symbol table?
• module kallsyms: requires type information and symbol information for vmlinux already. I don't think it's too picky about which source: kallsyms/ELF symtab should be fine, same with BTF/DWARF.
• module BTF: requires that vmlinux BTF is loaded, because the type definitions are built on top of the core definitions in vmlinux. Also, vmlinux symbols would almost certainly be needed to locate things.

[marxin]

Great, I just wanted to note that I would like to see kallsyms support when it comes to symbol lookup! Can you @brenns10 rebase the branch? I would like to test it..

I noticed crash can identify operations variable of block devices:

crash> dev
...
BLKDEV    NAME                GENDISK      OPERATIONS      
 259      blkext              (none)     
  11      sr             ffff9dcb85813800  sr_bdops
 253      virtblk        ffff9dcb87502600  virtblk_fops
 254      device-mapper       (none)     

where virtblk_fops is:

crash> sym ffffffffc04513c0
ffffffffc04513c0 (?) virtblk_fops [virtio_blk]

[brenns10]

Hi @marxin, this branch is quite old. It actually has been partially rebased and improved already. Let me try to break down what the current state is.

1. Pluggable symbol finder API - allowing a custom symbol lookup function. This branch contains the commit: Refactor ELF symbol search into Symbol Finder API. I've rebased that in a different PR. Plus, that same PR adds the ability for Python symbol finders to be implemented, which is a big win for me: much easier for prototyping.
2. This PR contains hooks that look at the vmcoreinfo note and grab some data, and then another hook that automatically add the kallsyms finder. I don't like that API, since Omar's module handling APIs will be better and give us more control over which debuginfo is loaded, so my hooks are quite a hack. The ideal case would be that kallsyms would be an option for kernel debuginfo. In the meantime, while it wouldn't be hard to rebase these hooks, I think it might be better for this symbol finder to be exposed to the Python API. With the newer PR (Pluggable Symbol finder API, with Python support #241) the vmcoreinfo is available to Python code, so you could just do something like prog.add_symbol_finder(drgn.kallsyms_vmlinux_symbol_finder(prog["VMCOREINFO"])).

So to sum up, what I'd like to do for this is:

1. Rebase Pluggable Symbol finder API, with Python support #241 - though that's quite new, it would be nice to base it on the latest and greatest.
2. Take the kallsyms finder that was implemented here, and add the Python bindings to configure it, rather than the weird hooks.
3. I'll also take a look at what python-based symbol finders I've implemented.

I noticed crash can identify operations variable of block devices:

Heh, crash has a few tricks up its sleeve. If you look at struct module it has a module_kallsyms that is significantly easier to parse than the vmlinux stuff. I'm not sure if crash bothers with that... There's also syms, gpl_syms fields, which are only for exported symbols of the module. Unfortunately, those fields don't contain all the ELF data for the symbols. I think the symbol you're looking at there is exported by the module and so crash got its info from there. Notice that crash doesn't know the symbol kind (data, function, etc).

Finally - do you want an invite to the Linux Kernel Debug slack? Some lower-latency drgn development discussions can happen there, and there's some community from outside the drgn world (libkdumpfile and crash folks, as well as Delphix/sdb folks, and others who are interested in this stuff). Essentially the same community as the monthly meetings (which are currently on hold for a couple months). If so, just shoot me an email via the commit author email from any of my branches/commits, and I'll get you setup :)

[marxin]

So to sum up, what I'd like to do for this is:

1. Rebase Pluggable Symbol finder API, with Python support, + vmcoreinfo goodies #241 - though that's quite new, it would be nice to base it on the latest and greatest.
2. Take the kallsyms finder that was implemented here, and add the Python bindings to configure it, rather than the weird hooks.
3. I'll also take a look at what python-based symbol finders I've implemented.

Hey. This sounds like a feasible solution one can use in the future. Actually, it's nothing super-important, I was only interested where crash founds the symbol name and I realized it's something provided by kallsysm.

Finally - do you want an invite to the Linux Kernel Debug slack? Some lower-latency drgn development discussions can happen there, and there's some community from outside the drgn world (libkdumpfile and crash folks, as well as Delphix/sdb folks, and others who are interested in this stuff). Essentially the same community as the monthly meetings (which are currently on hold for a couple months). If so, just shoot me an email via the commit author email from any of my branches/commits, and I'll get you setup :)

Thank you for the offer! However, my interest in drgn is only a side project and I'll be leaving my employer soon. On the other hand, there are my current colleagues @tehcaster and @Werkov who are interested in drgn and itss possible use as a crash replacement for the future.

[brenns10]

@marxin you can check out the rebased branch, the one which contains both the symbol finder API, and the kallsyms symbol finder API, in my kallsyms_finder branch. The full overview of everything is given in my (newly updated) issue #176.

For this pull request, I'll close it for two reasons. First, I've rebased the branch and now this one is outdated. Second, whatever happens with the symbol API, kallsyms finders, and CTF implementation described there... it all depends on the module API rework which is nearing completion. So the final implementation won't look like the branch here. I'm closing my PRs and just describing the current status on isuse #176 with links to current branches. Hopefully that will generate less noise in the issue / PR board.

[brenns10]

Oh and @tehcaster and @Werkov, please do feel free to reach out to me (see the committer email in one of my branches) and I'll gladly add you to the slack. I strongly feel that we all should be working together on debugging tools :)