ModulePasses with Auto Registration

[ghost]

In case you were interested, I was able to figure this out without having to do the other method of registering the pass following what was done here.

As you can see, trying to run the module pass before the other optimizations causes it generate segmentation faults/assertion faults depending on the method. Also, PassManagerBuilder::EP_ModuleOptimizerEarly worked as well. I'm assuming the rest of the options should work since they are working within the module. Here is my example that uses your basic framework from your examples.

Important Content:

 static RegisterStandardPasses**
    RegisterMyPass(PassManagerBuilder::EP_ModuleOptimizerEarly, registerSkeletonPass);

static RegisterStandardPasses
    RegisterMyPass0(PassManagerBuilder::EP_EnabledOnOptLevel0, registerSkeletonPass);

Source Code:

#include "llvm/Support/raw_ostream.h"
#include "llvm/ADT/Statistic.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/LegacyPassManager.h"
#include "llvm/IR/Module.h"
#include "llvm/Support/Debug.h"
#include "llvm/Transforms/IPO/PassManagerBuilder.h"

using namespace llvm;

namespace {
    struct SkeletonPass : public ModulePass {
        static char ID;
        SkeletonPass() : ModulePass(ID) { }

        virtual bool runOnModule(Module &M) override;
    };
}

char SkeletonPass::ID = 0;

bool SkeletonPass::runOnModule(Module &M) {
    errs() << "In module called: " << M.getName() << "!\n";

    for (auto &F : M) {
        F.dump();

        for (auto &B : F) {
            B.dump();

            for (auto &I : B) {
                I.dump();
            }
        }
    }

    return false;
}

//Automatically enable the pass.
//http://adriansampson.net/blog/clangpass.html
static void registerSkeletonPass(const PassManagerBuilder &,
                     legacy::PassManagerBase &PM) {
    PM.add(new SkeletonPass());
}

static RegisterStandardPasses
    RegisterMyPass(PassManagerBuilder::EP_ModuleOptimizerEarly, registerSkeletonPass);

static RegisterStandardPasses
    RegisterMyPass0(PassManagerBuilder::EP_EnabledOnOptLevel0, registerSkeletonPass);

[sampsyo]

Cool! So, to clarify, you just needed to choose the right extension point (EP_*) to make a module pass work? Seems right to me.

[ghost]

Correct, with EP_EarlyAsPossible I was getting Segmentation faults.

[sampsyo]

Yep, that matches my experience—some EPs are for function passes and some are for module passes, but this isn't well documented.

[ghost]

I've been working on it for the last day banging my head, so I'm glad I was able to figure it out. I'm assuming pass hierarchy goes something like this. So Function Passes don't even begin to run until after all of the Module Passes have been performed, and at the point at a point where LLVM can successfully run the pass. When it's classified as EarlyAsPossible for a module, it must place it in front of some setup required by LLVM to run.

Passes:
0. Default module passes that must be performed

1. Module
2. Function
3. BB
4. ...

[sampsyo]

I'm recalling purely from memory, but here's how I remember it being structured: there are ModulePassManagers and FunctionPassManagers. The registration has a switch on the EP flag, and some EPs map to one kind of pass manager and some map to the other. This mapping does no type checking to ensure that only ModulePasses get registered with ModulePassManagers and so on. Therefore, it's possible to choose the "wrong" kind of EP and get registered with the wrong pass manager. Then, you'll end up (for example) with a FunctionPassManager trying to call runOnFunction on your ModulePass, which of course doesn't exist.

[ghost]

Oh okay, that makes more sense. Well either way, I hope this can help others that may come across your examples. There are tons of FunctionPass code out there that is explained well, not so much for ModulePass.

[sampsyo]

Indeed; thanks for pointing it out! I'll leave this issue open in the hopes that other Google searchers will find it.

[PaoloS02]

Yes, it helped a lot, thank you!

[danpcampbell]

I am attempting to use this approach with LLVM 6.0. I get segmentation faults when loading the .so generated by this code, both in clang and in opt. Has the appropriate path been changed in 6?

[sampsyo]

Good question! I don't know what could be wrong—I know people in other threads have successfully used (prereleases of LLVM 6.0). Please let us know if you find the root cause of the segfault.

[danpcampbell]

will report back. It's almost certainly something I am doing wrong.

[minad]

It seems the segfault happens due to premature unloading of the shared object. It works for me if I compile the plugin with -Wl,-znodelete. This linker flag prevents unloading the shared object.

[LeadroyaL]

In case you were interested, I was able to figure this out without having to do the other method of registering the pass following what was done here.

static RegisterStandardPasses
    RegisterMyPass(PassManagerBuilder::EP_ModuleOptimizerEarly, registerSkeletonPass);

static RegisterStandardPasses
    RegisterMyPass0(PassManagerBuilder::EP_EnabledOnOptLevel0, registerSkeletonPass);

It works for me, now I can use ModulePass. Thanks for afl-fuzz!

[sampsyo]

I spent a while looking into this today. To be specific, the problem looks like this on my system, at the moment:

$ /usr/local/opt/llvm/bin/clang -fplugin=build/skeleton/libSkeletonPass.so foo.c
I saw a function called foo!
Stack dump:
0.	Program arguments: /usr/local/Cellar/llvm/7.0.1/bin/clang-7 -cc1 -triple x86_64-apple-macosx10.14.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name foo.c -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -masm-verbose -munwind-tables -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 409.12 -resource-dir /usr/local/Cellar/llvm/7.0.1/lib/clang/7.0.1 -fdebug-compilation-dir /Users/asampson/cu/llvm-pass-skeleton -ferror-limit 19 -fmessage-length 80 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fobjc-runtime=macosx-10.14.0 -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -load build/skeleton/libSkeletonPass.so -o /var/folders/xq/1tgxc1mj0c75vn44tf_6q96c0000gn/T/foo-1708e5.o -x c foo.c 
0  clang-7                  0x0000000109362adc llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 37
1  clang-7                  0x0000000109362ede SignalHandler(int) + 192
2  libsystem_platform.dylib 0x00007fff7ce58b3d _sigtramp + 29
3  libsystem_platform.dylib 0x00007ffee784eb38 _sigtramp + 1788829720
4  clang-7                  0x000000010905f74e llvm::object_deleter<llvm::SmallVector<std::__1::pair<llvm::PassManagerBuilder::ExtensionPointTy, std::__1::function<void (llvm::PassManagerBuilder const&, llvm::legacy::PassManagerBase&)> >, 8u> >::call(void*) + 22
5  clang-7                  0x0000000109320d85 llvm::llvm_shutdown() + 53
6  clang-7                  0x0000000109318c6d llvm::InitLLVM::~InitLLVM() + 15
7  clang-7                  0x00000001083b0bf7 main + 141
8  libdyld.dylib            0x00007fff7cc6ded9 start + 1
clang-7: error: unable to execute command: Segmentation fault: 11
clang-7: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 7.0.1 (tags/RELEASE_701/final)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm/bin
clang-7: note: diagnostic msg: PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
clang-7: error: unable to execute command: Segmentation fault: 11
clang-7: note: diagnostic msg: Error generating preprocessed source(s).

That is, everything goes fine until LLVM tries to shut itself down. The issue does indeed seem to be that recent version of LLVM unloads our shared library before shutting down LLVM. Then, when LLVM does shut down, it tries to free the memory for the pass registration functions, including our registerSkeletonPass, which is now a dangling pointer. The vector of functions in PassManagerBuilder that is populated using our RegisterStandardPasses is the thing that's crashing on destruction.

This problem has been reported to the LLVM bug tracker a few times, but hasn't gotten much traction:
https://bugs.llvm.org/show_bug.cgi?id=34573
https://bugs.llvm.org/show_bug.cgi?id=39321
https://bugs.llvm.org/show_bug.cgi?id=36183

Using -znodelete is a reasonable workaround, but this does seem like LLVM itself should fix. And that option doesn't seem to be available on my (macOS 10.14) linker.

[parth-shel]

Hi! Were you able to find a workaround with the -z flag to the linker for macOS?
I'm trying to dig into this, and I found this, but I don't really want to get into the LLVM code to patch this.

The TLDR is:
"Libraries should export initialization and cleanup routines using the gcc attribute((constructor)) and attribute((destructor)) function attributes. See the gcc info pages for information on these. Constructor routines are executed before dlopen returns (or before main() is started if the library is loaded at load time). Destructor routines are executed before dlclose returns (or after exit() or completion of main() if the library is loaded at load time)."

[sampsyo]

Nope, no current solution! If you find anything good (or can bring it to the attention of the LLVM developers), please let us know.

[mattgreen]

Dug into this issue a bit, here's what I found using LLVM 9 installed via Homebrew.

1. If you are okay using the new PassManager infrastructure to run your passes via opt, then just use that on macOS and ignore the rest of the message.

2. If you want to run your pass via clang, then you can use the legacy PassManager with a simple change: insert DYLD_INSERT_LIBRARIES=path/to/dylib before your clang/opt invocation. This preloads the pass shared library before starting the clang/opt. When clang/opt eventually request to unload the library, they will decrement the ref count of the shared library, but the OS won't actually unload it, as the preload's ref is still outstanding. This prevents the crash at shutdown.

[sampsyo]

That's a neat idea, @mattgreen! Cool workaround.

[minad]

It should be possible for the shared object to dlopen itself in a constructor such that the refcount is increased. Did someone try this? But I wonder why this hasn't been fixed in upstream llvm yet.

[mattgreen]

@minad I like that idea, but I'm not sure of an easy way to get the absolute path of the currently executing shared library on macOS. If you have any ideas please share!

[minad]

@mattgreen I have no idea about mac. For me things work if I use the nodelete linker flag. But I am pretty sure it is possible on Mac to iterate over the loaded shared objects/the address space of the current process. On Linux there is dl_iterate_phdr or you could access /proc/self/maps.

[minad]

But I am not sure if it is worth the effort to implement such a more sophisticated workaround. Llvm should be fixed instead.

[mattgreen]

Agree. The new PassManager + PassPlugin API doesn't seem to have these issues, I think that's why you're seeing it not be addressed. However, adoption of the new PassManager has been slow. Clang still has it gated behind -fexperimental-new-pass-manager IIRC, and I'm not sure opt is even using it yet.

[minad]

@mattgreen How does the new pass manager work? Can we keep all the things as is for the plugin and simply pass -fexperimental-new-pass-manager to clang and things magically start working without znodelete?

[mattgreen]

At least with clang-9, you can force it to load your shared library via the new pass manager:

$ clang-9 -fexperimental-new-pass-manager -fpass-plugin=myplugin.so a.c

clang will call llvmGetPassPluginInfo() and give you a chance to register your pass using the new PassPlugin API. Registering your passes in this way avoids the crashing issue due to the fact that LLVM is not holding onto a std::function allocated by the shared object after the shared object is unloaded.

I haven't used this myself; I'm content using the DYLD_INSERT_LIBRARIES hack with opt right now. I'm not sure if all of the extension points are supported yet with the new PassManager, so that's one reason why I'm waiting.

See this snippet in llvm-tutor for an example of how to use the new PassPlugin API.

[sampsyo]

There's also more on the "pass plugin" registration system in the docs:
http://llvm.org/docs/WritingAnLLVMPass.html#building-pass-plugins

[EliaGeretto]

The bug in RegisterStandardPasses should be fixed in LLVM 10, I submitted a patch that was accepted some days ago. It was dependent on the type of LLVM build and optimization level being used.

This is the commit, for reference: llvm/llvm-project@52c1d20

A full analysis of the problem can be found here: https://bugs.llvm.org/show_bug.cgi?id=39321

[sampsyo]

Wow! That's awesome! Thank you for tracking this down, @EliaGeretto, and driving the effort to get it fixed. Wahoo!

[banach-space]

Super happy to see this being fixed, thank you @EliaGeretto !

There's also more on the "pass plugin" registration system in the docs:
http://llvm.org/docs/WritingAnLLVMPass.html#building-pass-plugins

@sampsyo That section (and the functionality that it refers to) has only landed recently (commit), so we still have to wait for LLVM 10 to be released. Also, AFAIK, that mechanism is only for linking plugins statically (rather than dynamically). Either way, it's fantastic to have it there!

[minad]

Also, AFAIK, that mechanism is only for linking plugins statically

What does this mean? The whole issue is about dynamic objects being unloaded prematurely leading to segfaults.

In my case it would be really nice if dynamic plugins work properly such that stock clang can be used.

[banach-space]

Also, AFAIK, that mechanism is only for linking plugins statically

What does this mean? The whole issue is about dynamic objects being unloaded prematurely leading to segfaults.

That was a bit off-topic, sorry! What I was trying to say is that the mechanism described in the docs (http://llvm.org/docs/WritingAnLLVMPass.html#building-pass-plugins) doesn't really change how we develop and use plugins that are loaded dynamically. However, the introduction of LLVM_${NAME}_LINK_INTO_TOOLS simplifies immensely how statically linked plugins are developed.