clang_notes.txt

----------------------------
NOTES ON BUILDING WITH CLANG 
----------------------------

Issues encountered trying to build NGSolve:

The GNU Standard C++ Library for the Centos 6.10 system is old.  NGSolve
need libstdc++ to have GLIBCXX up to 3.4.28 and CXXABI up to 1.3.12.  To 
support this, I first put all the gcc-9.2.0 library locations in the
LD_LIBRARY_PATH in the clang-11.1.0 module file.

Next issue: Clang selects the runtime with the old /usr/lib/libstdc++ and then
cannot compile any c++17 constructs.  However, we can use the --gcc-toolchain
clang flag to tell clang to use gcc-9.2.0 as the runtime.  This fixed the
compile errors (missing c++17 definitions)

But I now get linker errors when building the netgen binary.  I think the gnu
linker ld is being invoked, and not llvm's lld because

clang++ -o test3 --gcc-toolchain=/vol/apps/gcc/gcc-9.2.0 -std=c++17 test3.cpp
readelf --string-dump .comment test3

yielded:

 String dump of section '.comment':
 [     0]  GCC: (GNU) 4.8.5 20150623 (Red Hat 4.8.5-39)
 [    2d]  GCC: (GNU) 9.2.0
 [    3e]  clang version 11.1.0

According to the LLD docs, LLD logs its name and version number to a .comment
section in the output.

with $topdir=$HOME/local/llvm-project/install.
I tried adding these to LD_LIBRARY_PATH:
/usr/lib64, /lib64, /usr/lib/gcc/x86_64-redhat-linux/4.8.2,
$topdir/lib/x86_64-unknown-linux-gnu/c++
Removing $topdir/lib causes errors right away (no working compiler)

I tried to troubleshoot this issue using ldd.  Note that ldd output depends on
context.  If the gcc-9.2.0 module is not loaded, the output of lld with
libnglib.so, which built correctly using both gcc and clang, shows many mesages
like "/usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found."  When
appropriate modules have been loaded, these errors disappear and identical
standard library references are obtained.

I haven't been able to work out why the netgen binary fails to link.  All the
messages seem to refer to undefined c++ 11 functions.  This indicates that the
gcc-9.2.0 installation isn't self-contained; but the libstdc++ libraries that
define these objects should still be in the search path.  My guess is that
setting --gcc-toolchain to gcc-9.2.0, somehow breaks the ability of the linker
to find these c++ 11 functions, but it's not clear.

This post:
https://stackoverflow.com/questions/847179/multiple-glibc-libraries-on-a-single-host
has quite a bit of information on this sort of linker issue but it's mostly
over my head.  Some talk of LD_PRELOAD instead of LD_LIBRARY_PATH possibly
working.  This gets tricky, as I recall, the order in which the libraries are
loaded can break a build.

I thought about trying to construct a simple example that would reproduce the
linker error, but the netgen source was a rabbit hole.  It still might be
possible to reproduce it by creating two object files, one referencing C++17
libraries and the other referencing C++11 libraries, then explicitly linking
these to a library or executable.  This might shed some light on the problem.

Comparing the verbose output of the clang and gcc builds, the main item that
looked out of place was the presence of
/home/ddrake/local/llvm-project/install/lib/x86_64-unknown-linux-gnu/c++ in the
library search path.  This path contains libc++.so.1, which I thought might
somehow be the issue, but moving it last in the search path didn't help.  I
don't actually think that any of the libraries there would be used by the build
anyway.

I tried to build libstdc++.so and libcxxabi.so using LLVM.  I think this was
neither necessary nor a good idea.  I first tried to do the build using the
clang/clang++ compilers, but got a message that they didn't support enough
standard library.  This was before I knew how to specify the gcc toolchain.
I then tried to build using gcc, but the linker process was always
killed right before completing the build of LLVM.  I tried to streamline the
build as much as possible but it still exceeds memory right after building the
target llvm-reduce.  This is peculiar, since the reduced build is basically
identical to the build used for clang, etc.  and the file sizes are small.
I haven't tried to build this on my local Ubuntu machine yet.