[clang] clang driver fails to find headers/libraries when specified --gcc-install-dir or --gcc-toolchain or --sysroot to a cross-compiler directory

[lbmeng]

I am trying to use clang 18 to cross compile a program for Arm 32-bit target.
The clang 18 was installed from the official LLVM Ubuntu repository following the instructions here.

The GCC toolchain I was using is from bootlin.

The rough GCC toolchain directory layout is:

/path/to/bootlin/toolchain
├── arm-buildroot-linux-gnueabihf
│   ├── bin
│   ├── include
│   │   └── c++
│   ├── lib
│   └── sysroot
│       ├── bin
│       ├── dev
│       ├── etc
│       ├── lib
│       ├── lib32 -> lib
│       ├── media
│       ├── mnt
│       ├── opt
│       ├── proc
│       ├── root
│       ├── run
│       ├── sbin
│       ├── sys
│       ├── tmp
│       ├── usr
│       └── var
├── bin
├── etc
├── include
├── lib
│   ├── gcc
│   │   └── arm-buildroot-linux-gnueabihf
...
├── lib64 -> lib
├── libexec
└── share

Compiling a simple program using:

$ clang-18 --target=arm-eabi -mcpu=cortex-a7 -mthumb -mfpu=neon -mfloat-abi=hard -Wall --gcc-toolchain=/path/to/bootlin/toolchain -c test.c -o test.o
clang-18: warning: argument unused during compilation: '--gcc-toolchain=/path/to/bootlin/toolchain' [-Wunused-command-line-argument]
test.c:1:10: fatal error: 'stdio.h' file not found
    1 | #include <stdio.h>
      |          ^~~~~~~~~
1 error generated.

clang-18 complains --gcc-toolchain is unused. Changing to use --gcc-install-dir does not make it better. The same unused argument warning was thrown. By looking at the documentation, the doc does not mention how exactly these 2 options affects what behavior of clang.

Changing to use--sysroot failed too. The sysroot value was obtained from the result of /path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-gcc --print-sysroot.

Then I found a workaround to help locate the header file by creating a symbolic link include to point to usr/include in the sysroot directory when using

/path/to/bootlin/toolchain/arm-buildroot-linux-gnueabihf/sysroot$ ln -s usr/include include

Then I removed -c to see if I can compile and link the program correctly, I got:

$ clang-18 --target=arm-eabi -mcpu=cortex-a7 -mthumb -mfpu=neon -mfloat-abi=hard -Wall --sysroot=/path/to/bootlin/toolchain/arm-buildroot-linux-gnueabihf/sysroot test.c -o test.o
ld.lld: error: unable to find library -lc
ld.lld: error: unable to find library -lm
ld.lld: error: unable to find library -lgcc
clang-18: error: ld.lld command failed with exit code 1 (use -v to see invocation)

I realized the new clang compiler driver defaults to use ld.lld, so I added -fuse-ld=/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld in the command line, which made clang to successfully locate libc and libm libraries, but not libgcc.

$ clang-18 --target=arm-eabi -mcpu=cortex-a7 -mthumb -mfpu=neon -mfloat-abi=hard -Wall --sysroot=/path/to/bootlin/toolchain/arm-buildroot-linux-gnueabihf/sysroot -fuse-ld=/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld test.c -o test.o
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: warning: library search path "/usr/lib/llvm-18/lib/clang/18/lib/baremetal" is unsafe for cross-compilation
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: cannot find -lgcc: No such file or directory
clang-18: error: ld.lld command failed with exit code 1 (use -v to see invocation)

The libgcc.a is located in the bootlin gcc toolchain directorty here:

$ /path/to/bootlin/toolchain$ find . -name libgcc.a
./lib/gcc/arm-buildroot-linux-gnueabihf/12.3.0/libgcc.a

So I added -L/path/to/bootlin/toolchain/lib/gcc/arm-buildroot-linux-gnueabihf/12.3.0 in the command line, then libgcc was located by clang, but the final linking procedure still complained:

$ clang-18 --target=arm-eabi -mcpu=cortex-a7 -mthumb -mfpu=neon -mfloat-abi=hard -Wall --sysroot=/path/to/bootlin/toolchain/arm-buildroot-linux-gnueabihf/sysroot -fuse-ld=/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld test.c -L/path/to/bootlin/toolchain/lib/gcc/arm-buildroot-linux-gnueabihf/12.3.0 -o test.o
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: warning: library search path "/usr/lib/llvm-18/lib/clang/18/lib/baremetal" is unsafe for cross-compilation
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: warning: cannot find entry symbol _start; defaulting to 00010140
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: /path/to/bootlin/toolchain/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libc.a(printf.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr1'
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: /path/to/bootlin/toolchain/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libc.a(getsysstats.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr0'
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: /path/to/bootlin/toolchain/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libc.a(getsysstats.o):(.ARM.exidx+0x8): undefined reference to `__aeabi_unwind_cpp_pr1'
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: /path/to/bootlin/toolchain/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libc.a(madvise.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr0'
...

I gave up here. So what is the expected way of using clang to cross-compile a program and using libraries from a gcc cross-toolchain?

[llvmbot]

@llvm/issue-subscribers-clang-driver

Author: Bin Meng (lbmeng)

I am trying to use clang 18 to cross compile a program for Arm 32-bit target. The clang 18 was installed from the official LLVM Ubuntu repository following the instructions [here](https://apt.llvm.org/).

The GCC toolchain I was using is from bootlin.

The rough GCC toolchain directory layout is:

/path/to/bootlin/toolchain
├── arm-buildroot-linux-gnueabihf
│   ├── bin
│   ├── include
│   │   └── c++
│   ├── lib
│   └── sysroot
│       ├── bin
│       ├── dev
│       ├── etc
│       ├── lib
│       ├── lib32 -> lib
│       ├── media
│       ├── mnt
│       ├── opt
│       ├── proc
│       ├── root
│       ├── run
│       ├── sbin
│       ├── sys
│       ├── tmp
│       ├── usr
│       └── var
├── bin
├── etc
├── include
├── lib
│   ├── gcc
│   │   └── arm-buildroot-linux-gnueabihf
...
├── lib64 -> lib
├── libexec
└── share

Compiling a simple program using:

$ clang-18 --target=arm-eabi -mcpu=cortex-a7 -mthumb -mfpu=neon -mfloat-abi=hard -Wall --gcc-toolchain=/path/to/bootlin/toolchain -c test.c -o test.o
clang-18: warning: argument unused during compilation: '--gcc-toolchain=/path/to/bootlin/toolchain' [-Wunused-command-line-argument]
test.c:1:10: fatal error: 'stdio.h' file not found
    1 | #include <stdio.h>
      |          ^~~~~~~~~
1 error generated.

clang-18 complains --gcc-toolchain is unused. Changing to use --gcc-install-dir does not make it better. The same unused argument warning was thrown. By looking at the documentation, the doc does not mention how exactly these 2 options affects what behavior of clang.

Changing to use--sysroot failed too. The sysroot value was obtained from the result of /path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-gcc --print-sysroot.

Then I found a workaround to help locate the header file by creating a symbolic link include to point to usr/include in the sysroot directory when using

/path/to/bootlin/toolchain/arm-buildroot-linux-gnueabihf/sysroot$ ln -s usr/include include

Then I removed -c to see if I can compile and link the program correctly, I got:

$ clang-18 --target=arm-eabi -mcpu=cortex-a7 -mthumb -mfpu=neon -mfloat-abi=hard -Wall --sysroot=/path/to/bootlin/toolchain/arm-buildroot-linux-gnueabihf/sysroot test.c -o test.o
ld.lld: error: unable to find library -lc
ld.lld: error: unable to find library -lm
ld.lld: error: unable to find library -lgcc
clang-18: error: ld.lld command failed with exit code 1 (use -v to see invocation)

I realized the new clang compiler driver defaults to use ld.lld, so I added -fuse-ld=/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld in the command line, which made clang to successfully locate libc and libm libraries, but not libgcc.

$ clang-18 --target=arm-eabi -mcpu=cortex-a7 -mthumb -mfpu=neon -mfloat-abi=hard -Wall --sysroot=/path/to/bootlin/toolchain/arm-buildroot-linux-gnueabihf/sysroot -fuse-ld=/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld test.c -o test.o
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: warning: library search path "/usr/lib/llvm-18/lib/clang/18/lib/baremetal" is unsafe for cross-compilation
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: cannot find -lgcc: No such file or directory
clang-18: error: ld.lld command failed with exit code 1 (use -v to see invocation)

The libgcc.a is located in the bootlin gcc toolchain directorty here:

$ /path/to/bootlin/toolchain$ find . -name libgcc.a
./lib/gcc/arm-buildroot-linux-gnueabihf/12.3.0/libgcc.a

So I added -L/path/to/bootlin/toolchain/lib/gcc/arm-buildroot-linux-gnueabihf/12.3.0 in the command line, then libgcc was located by clang, but the final linking procedure still complained:

$ clang-18 --target=arm-eabi -mcpu=cortex-a7 -mthumb -mfpu=neon -mfloat-abi=hard -Wall --sysroot=/path/to/bootlin/toolchain/arm-buildroot-linux-gnueabihf/sysroot -fuse-ld=/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld test.c -L/path/to/bootlin/toolchain/lib/gcc/arm-buildroot-linux-gnueabihf/12.3.0 -o test.o
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: warning: library search path "/usr/lib/llvm-18/lib/clang/18/lib/baremetal" is unsafe for cross-compilation
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: warning: cannot find entry symbol _start; defaulting to 00010140
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: /path/to/bootlin/toolchain/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libc.a(printf.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr1'
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: /path/to/bootlin/toolchain/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libc.a(getsysstats.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr0'
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: /path/to/bootlin/toolchain/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libc.a(getsysstats.o):(.ARM.exidx+0x8): undefined reference to `__aeabi_unwind_cpp_pr1'
/path/to/bootlin/toolchain/bin/arm-buildroot-linux-gnueabihf-ld: /path/to/bootlin/toolchain/bin/../arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libc.a(madvise.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr0'
...

I gave up here. So what is the expected way of using clang to cross-compile a program and using libraries from a gcc cross-toolchain?