Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Link LLVM-based tools dynamically to LLVM #397

Closed
wants to merge 3 commits into from

Conversation

alexcrichton
Copy link
Collaborator

I noticed that #388 increased the size of the Windows distribution by 200M+ and my guess for that is that all the symlinks which point to clang get copied as real files (due to symlinks not always working on Windows). To help counteract that this commit enables the LLVM_LINK_LLVM_DYLIB=ON option when building LLVM and Clang. That should build a libLLVM.so which helps reduce the size of all the tools since they no longer all statically link LLVM. Locally the size of a full build is ~100M less as a result of this change. On Windows where all the binaries are copied around it should hopefully help much more.

I've additionally taken a leaf out of rust-lang/rust's book of building LLVM to pass the LLVM_VERSION_SUFFIX option here too. That I believe helps avoid libLLVM.so conflicting with a system-installed libLLVM.so by accident by ensuring there's a suffix present on the binaries built for wasi-sdk.

I noticed that WebAssembly#388 increased the size of the Windows distribution by
200M+ and my guess for that is that all the symlinks which point to
clang get copied as real files (due to symlinks not always working on
Windows). To help counteract that this commit enables the
`LLVM_LINK_LLVM_DYLIB=ON` option when building LLVM and Clang. That
should build a `libLLVM.so` which helps reduce the size of all the tools
since they no longer all statically link LLVM. Locally the size of a
full build is ~100M less as a result of this change. On Windows where
all the binaries are copied around it should hopefully help much more.

I've additionally taken a leaf out of rust-lang/rust's book of building
LLVM to pass the `LLVM_VERSION_SUFFIX` option here too. That I believe
helps avoid `libLLVM.so` conflicting with a system-installed
`libLLVM.so` by accident by ensuring there's a suffix present on the
binaries built for wasi-sdk.
@@ -78,6 +78,8 @@ build/llvm.BUILT:
-DCLANG_LINKS_TO_CREATE="$(call join-with,;,$(CLANG_LINKS_TO_CREATE))" \
-DCMAKE_BUILD_TYPE=MinSizeRel \
-DLLVM_ENABLE_TERMINFO=OFF \
-DLLVM_LINK_LLVM_DYLIB=ON \
-DLLVM_VERSION_SUFFIX=-wasi-sdk \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that bin/clang-18 is now bin/clang-18-wasi-sdk? Or does it just apply the .so file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it's just the *.so files, from the installation folder I get:

 $ fd -- -wasi-sdk
lib/libLLVM-18-wasi-sdk.so
lib/libLLVM.so.18.1-wasi-sdk
lib/libclang-cpp.so.18.1-wasi-sdk

can confirm that bin/clang-18 is still named just that, no wasi-sdk

@@ -78,6 +78,8 @@ build/llvm.BUILT:
-DCLANG_LINKS_TO_CREATE="$(call join-with,;,$(CLANG_LINKS_TO_CREATE))" \
-DCMAKE_BUILD_TYPE=MinSizeRel \
-DLLVM_ENABLE_TERMINFO=OFF \
-DLLVM_LINK_LLVM_DYLIB=ON \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me if it works. I assume the rpath stuff in the binaries is setup so that they can find the .so files relative to themselves?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe so yeah, bin/clang has:

$ readelf -a ./bin/clang | rg -i runpath
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib]

I wasn't certain for other platforms so I was relying on CI here during make check to fail if there was an issue

@@ -118,6 +120,8 @@ build/llvm.BUILT:
install-objdump \
install-objcopy \
install-c++filt \
install-LLVM \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this install anything more than libLLVM ? e.g. does it also install headers?

Is this technically needed? Does install-clang, for example, not automatically also install libLLVM.so?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this I found that as-is without adding this libLLVM wasn't installed, so the lib dir was empty, meaning that all the binaries failed with:

$ ./bin/clang --version
./bin/clang: error while loading shared libraries: libclang-cpp.so.18.1-wasi-sdk: cannot open shared object file: No such file or directory

to see the missing deps I ran:

$ ldd bin/clang
        linux-vdso.so.1 (0x00007ffd18265000)
        libclang-cpp.so.18.1-wasi-sdk => not found
        libLLVM.so.18.1-wasi-sdk => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0511b62000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0511dc4000)

I wasn't sure how best to install these so I poked around the tab-completions of ninja install* and found install-{LLVM,clang-cpp} did the trick.

I ran ninja install-LLVM manually and it printed:

$ DESTDIR=$HOME/code/wasi-sdk/build ninja install-LLVM
[0/1] cd /home/alex/code/wasi-sdk/build/llvm/tools/llv...home/alex/code/wasi-sdk/build/llvm/cmake_install.cmake
-- Install configuration: "MinSizeRel"
-- Installing: /home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libLLVM.so.18.1-wasi-sdk
-- Set non-toolchain portion of runtime path of "/home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libLLVM.so.18.1-wasi-sdk" to "$ORIGIN/../lib"
-- Installing: /home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libLLVM.so
-- Creating libLLVM-18-wasi-sdk.so

and for clang-cpp:

 $ DESTDIR=$HOME/code/wasi-sdk/build ninja install-clang-cpp
[0/1] cd /home/alex/code/wasi-sdk/build/llvm/tools/clang/tools/clang-shlib && /home/alex/root/bin/cmake -DCMAKE_INSTALL_COMPONENT="clang-cpp" -P /home/alex/code/wasi-sdk/build/llvm/cmake_install.cmake
-- Install configuration: "MinSizeRel"
-- Installing: /home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libclang-cpp.so.18.1-wasi-sdk
-- Set non-toolchain portion of runtime path of "/home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libclang-cpp.so.18.1-wasi-sdk" to "$ORIGIN/../lib"
-- Installing: /home/alex/code/wasi-sdk/build/opt/wasi-sdk/lib/libclang-cpp.so

Sorry if this is a bit verbose, but I don't really have much of an idea of what I'm doing, I'm sort of just hitting the first thing that works and putting it here. I am by no means an expert in CMake, LLVM's build system, or how all this is supposed to interact.

My buest guest to your questions is:

Does this install anything more than libLLVM ? e.g. does it also install headers?

Judging from the above I think not, but I wouldn't say that with certainty.

Is this technically needed? Does install-clang, for example, not automatically also install libLLVM.so?

Apparently not, because if I left this out libLLVM.so wasn't installed.

I'm not sure if there's some other umbrella install-* target that does what we want though.

@@ -118,6 +120,8 @@ build/llvm.BUILT:
install-objdump \
install-objcopy \
install-c++filt \
install-LLVM \
install-clang-cpp \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this line install? cpp I assume refers to the pre-processor rather than clang++?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I commented a bit more above on this, but I thought clang-cpp was the preprocessor as well. Apparently though this deals with libclang-cpp.so and may just be slightly-oddly-named.

@alexcrichton
Copy link
Collaborator Author

Well, unfortunately, I'm stuck and I don't know how to fix CI. It's not a spurious error by any means, but here's what I've learned so far

  • The error is Windows-only.
  • The build failure happens on when using llvm-ar.exe, just built from LLVM source, to create a compiler-rt archive.
  • The compiler-rt archive does not exist, and it's not supposed to, and the qc flags passed to llvm-ar should create it.
  • Here llvm-ar should realize that the file could not be opened, but the error should equate to errc::no_such_file_or_directory.
  • In debugging I'm seeing the error's value method and message method return the same value errc::no_such_file_or_directory.

Basically I don't know why the error coming out of opening the file looks the same as errc::no_such_file_or_directory from what I can tell but does not equate. That means that llvm-ar is not realizing that it's an error it should handle later, and that fails the build.

How this is related to dynamic linking, I have no idea. I also don't know enough C++ to know really what std::error_code even is. To complicate matters there's also MinGW in the mix and I don't know how that plays in here. For example the error's value is 2 which is ENOENT on Linux but on Windows native APIs I think ERROR_NOT_FOUND would be used for this which does not have a value of 2.

I have not tried to reproduce this locally, mostly because replicating a CI environment for Windows is notoriously difficult once MinGW and msys2 are thrown into the mix.

So given all that, I've no idea what's happening here. Everything seems like it should work but for some reason it's not, and how that all relates to dynamic linking of C++ code is beyond me.

alexcrichton added a commit to alexcrichton/wasi-sdk that referenced this pull request Jul 12, 2024
This is a revival of WebAssembly#397 but only applies the logic to non-Windows
platforms. Windows builds still seem to not work so Windows is not
improved as a result of this change.
alexcrichton added a commit that referenced this pull request Jul 12, 2024
This is a revival of #397 but only applies the logic to non-Windows
platforms. Windows builds still seem to not work so Windows is not
improved as a result of this change.
@alexcrichton alexcrichton deleted the dynamic-build branch July 31, 2024 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants