Link LLVM-based tools dynamically to LLVM

[alexcrichton]

I noticed that #388 increased the size of the Windows distribution by 200M+ and my guess for that is that all the symlinks which point to clang get copied as real files (due to symlinks not always working on Windows). To help counteract that this commit enables the LLVM_LINK_LLVM_DYLIB=ON option when building LLVM and Clang. That should build a libLLVM.so which helps reduce the size of all the tools since they no longer all statically link LLVM. Locally the size of a full build is ~100M less as a result of this change. On Windows where all the binaries are copied around it should hopefully help much more.

I've additionally taken a leaf out of rust-lang/rust's book of building LLVM to pass the LLVM_VERSION_SUFFIX option here too. That I believe helps avoid libLLVM.so conflicting with a system-installed libLLVM.so by accident by ensuring there's a suffix present on the binaries built for wasi-sdk.

[sbc100]

Does this mean that bin/clang-18 is now bin/clang-18-wasi-sdk? Or does it just apply the .so file?

[alexcrichton]

Looks like it's just the *.so files, from the installation folder I get:

 $ fd -- -wasi-sdk
lib/libLLVM-18-wasi-sdk.so
lib/libLLVM.so.18.1-wasi-sdk
lib/libclang-cpp.so.18.1-wasi-sdk

can confirm that bin/clang-18 is still named just that, no wasi-sdk

[sbc100]

Sounds good to me if it works. I assume the rpath stuff in the binaries is setup so that they can find the .so files relative to themselves?

[alexcrichton]

I believe so yeah, bin/clang has:

$ readelf -a ./bin/clang | rg -i runpath
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib]

I wasn't certain for other platforms so I was relying on CI here during make check to fail if there was an issue

[sbc100]

Does this install anything more than libLLVM ? e.g. does it also install headers?

Is this technically needed? Does install-clang, for example, not automatically also install libLLVM.so?

[alexcrichton]

For this I found that as-is without adding this libLLVM wasn't installed, so the lib dir was empty, meaning that all the binaries failed with:

$ ./bin/clang --version
./bin/clang: error while loading shared libraries: libclang-cpp.so.18.1-wasi-sdk: cannot open shared object file: No such file or directory

to see the missing deps I ran:

$ ldd bin/clang
        linux-vdso.so.1 (0x00007ffd18265000)
        libclang-cpp.so.18.1-wasi-sdk => not found
        libLLVM.so.18.1-wasi-sdk => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0511b62000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0511dc4000)

I wasn't sure how best to install these so I poked around the tab-completions of ninja install* and found install-{LLVM,clang-cpp} did the trick.

I ran ninja install-LLVM manually and it printed:

$ DESTDIR=$HOME/code/wasi-sdk/build ninja install-LLVM
[0/1] cd /home/alex/code/wasi-sdk/build/llvm/tools/llv...home/alex/code/wasi-sdk/build/llvm/cmake_install.cmake
-- Install configuration: "MinSizeRel"
-- Installing: /home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libLLVM.so.18.1-wasi-sdk
-- Set non-toolchain portion of runtime path of "/home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libLLVM.so.18.1-wasi-sdk" to "$ORIGIN/../lib"
-- Installing: /home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libLLVM.so
-- Creating libLLVM-18-wasi-sdk.so

and for clang-cpp:

 $ DESTDIR=$HOME/code/wasi-sdk/build ninja install-clang-cpp
[0/1] cd /home/alex/code/wasi-sdk/build/llvm/tools/clang/tools/clang-shlib && /home/alex/root/bin/cmake -DCMAKE_INSTALL_COMPONENT="clang-cpp" -P /home/alex/code/wasi-sdk/build/llvm/cmake_install.cmake
-- Install configuration: "MinSizeRel"
-- Installing: /home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libclang-cpp.so.18.1-wasi-sdk
-- Set non-toolchain portion of runtime path of "/home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libclang-cpp.so.18.1-wasi-sdk" to "$ORIGIN/../lib"
-- Installing: /home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libclang-cpp.so

Sorry if this is a bit verbose, but I don't really have much of an idea of what I'm doing, I'm sort of just hitting the first thing that works and putting it here. I am by no means an expert in CMake, LLVM's build system, or how all this is supposed to interact.

My buest guest to your questions is:

Does this install anything more than libLLVM ? e.g. does it also install headers?

Judging from the above I think not, but I wouldn't say that with certainty.

Is this technically needed? Does install-clang, for example, not automatically also install libLLVM.so?

Apparently not, because if I left this out libLLVM.so wasn't installed.

I'm not sure if there's some other umbrella install-* target that does what we want though.

[sbc100]

What does this line install? cpp I assume refers to the pre-processor rather than clang++?

[alexcrichton]

I commented a bit more above on this, but I thought clang-cpp was the preprocessor as well. Apparently though this deals with libclang-cpp.so and may just be slightly-oddly-named.

[alexcrichton]

Well, unfortunately, I'm stuck and I don't know how to fix CI. It's not a spurious error by any means, but here's what I've learned so far

• The error is Windows-only.
• The build failure happens on when using llvm-ar.exe, just built from LLVM source, to create a compiler-rt archive.
• The compiler-rt archive does not exist, and it's not supposed to, and the qc flags passed to llvm-ar should create it.
• Here llvm-ar should realize that the file could not be opened, but the error should equate to errc::no_such_file_or_directory.
• In debugging I'm seeing the error's value method and message method return the same value errc::no_such_file_or_directory.

Basically I don't know why the error coming out of opening the file looks the same as errc::no_such_file_or_directory from what I can tell but does not equate. That means that llvm-ar is not realizing that it's an error it should handle later, and that fails the build.

How this is related to dynamic linking, I have no idea. I also don't know enough C++ to know really what std::error_code even is. To complicate matters there's also MinGW in the mix and I don't know how that plays in here. For example the error's value is 2 which is ENOENT on Linux but on Windows native APIs I think ERROR_NOT_FOUND would be used for this which does not have a value of 2.

I have not tried to reproduce this locally, mostly because replicating a CI environment for Windows is notoriously difficult once MinGW and msys2 are thrown into the mix.

So given all that, I've no idea what's happening here. Everything seems like it should work but for some reason it's not, and how that all relates to dynamic linking of C++ code is beyond me.