Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Different glibc version between aarch64 and x86 builders? #46186

Open
corneliusroemer opened this issue Mar 5, 2024 · 5 comments
Open

Different glibc version between aarch64 and x86 builders? #46186

corneliusroemer opened this issue Mar 5, 2024 · 5 comments
Labels
aarch64 Related to adding linux-aarch64 support

Comments

@corneliusroemer
Copy link
Member

I've tried to add aarch64 support to the nextclade recipe here #46185 which does binary repackaging.

I noticed that the aarch64 tests fail with:

15:17:28 BIOCONDA INFO (OUT) + nextclade --version
15:17:28 BIOCONDA INFO (OUT) nextclade: relocation error: nextclade: symbol __cxa_thread_atexit_impl, version GLIBC_2.18 not defined in file libc.so.6 with link time reference

The x86 tests don't fail - suggesting the version of glibc that comes with the aarch64 builder is lower than the one that comes with x86. Is that correct?

I know we should build from source - but repackaging has worked flawlessly to date. Others might run into the same issue.
@corneliusroemer corneliusroemer added the aarch64 Related to adding linux-aarch64 support label Mar 5, 2024
This was referenced Mar 5, 2024
Merged
Open
@martin-g
Copy link
Contributor

martin-g commented Mar 5, 2024

Both Docker images (quay.io/bioconda/bioconda-utils-build-env-cos7:latest and quay.io/bioconda/bioconda-utils-build-env-cos7-aarch64:latest) are based on CentOS 7 (7.9.2009)

aarch64:

[conda@26d3c04003f9 ~]$ ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[conda@26d3c04003f9 ~]$ ld --version
GNU ld version 2.27-44.base.el7_9.1
Copyright (C) 2016 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.


strings /usr/lib64/libc.so.6 | grep GLIBC
GLIBC_2.17
GLIBC_2.18
GLIBC_PRIVATE
gethostbyname2_r@@GLIBC_2.17
_IO_fopen@@GLIBC_2.17
_IO_file_finish@@GLIBC_2.17
_dl_starting_up@@GLIBC_PRIVATE
_IO_fgetpos@@GLIBC_2.17
getaliasent_r@@GLIBC_2.17
_IO_file_seekoff@@GLIBC_2.17
getrpcent_r@@GLIBC_2.17
getservbyname_r@@GLIBC_2.17
_rtld_global@@GLIBC_PRIVATE
_IO_file_init@@GLIBC_2.17
getservbyport_r@@GLIBC_2.17
__libc_enable_secure@@GLIBC_PRIVATE
posix_spawn@@GLIBC_2.17
tmpfile@@GLIBC_2.17
realpath@@GLIBC_2.17
_IO_fgetpos64@@GLIBC_2.17
getrpcbyname_r@@GLIBC_2.17
_rtld_global_ro@@GLIBC_PRIVATE
__pointer_chk_guard@@GLIBC_PRIVATE
getspent_r@@GLIBC_2.17
_IO_file_close_it@@GLIBC_2.17
sys_sigabbrev@@GLIBC_2.17
_IO_fsetpos@@GLIBC_2.17
_sys_nerr@@GLIBC_2.17
pthread_cond_timedwait@@GLIBC_2.17
_IO_do_write@@GLIBC_2.17
getnetbyname_r@@GLIBC_2.17
getpwnam_r@@GLIBC_2.17
_mcount@@GLIBC_2.18
gethostbyname_r@@GLIBC_2.17
_IO_file_setbuf@@GLIBC_2.17
pthread_cond_destroy@@GLIBC_2.17
getaliasbyname_r@@GLIBC_2.17
getprotoent_r@@GLIBC_2.17
getspnam_r@@GLIBC_2.17
getrpcbynumber_r@@GLIBC_2.17
nftw64@@GLIBC_2.17
getnetent_r@@GLIBC_2.17
_res@GLIBC_2.17
posix_spawnp@@GLIBC_2.17
_IO_popen@@GLIBC_2.17
pclose@@GLIBC_2.17
nftw@@GLIBC_2.17
_IO_file_xsputn@@GLIBC_2.17
_sys_errlist@@GLIBC_2.17
_IO_proc_open@@GLIBC_2.17
_IO_file_overflow@@GLIBC_2.17
_IO_fdopen@@GLIBC_2.17
getgrgid_r@@GLIBC_2.17
getnetbyaddr_r@@GLIBC_2.17
__res_maybe_init@GLIBC_PRIVATE
pthread_cond_signal@@GLIBC_2.17
h_errno@GLIBC_PRIVATE
_IO_file_attach@@GLIBC_2.17
_IO_file_sync@@GLIBC_2.17
getpwuid_r@@GLIBC_2.17
pthread_cond_wait@@GLIBC_2.17
_sys_siglist@@GLIBC_2.17
fopencookie@@GLIBC_2.17
pthread_cond_broadcast@@GLIBC_2.17
getpwent_r@@GLIBC_2.17
gethostbyaddr_r@@GLIBC_2.17
sched_setaffinity@@GLIBC_2.17
getgrent_r@@GLIBC_2.17
fnmatch@@GLIBC_2.17
getgrnam_r@@GLIBC_2.17
_IO_fclose@@GLIBC_2.17
pthread_cond_init@@GLIBC_2.17
regexec@@GLIBC_2.17
localeconv@@GLIBC_2.17
_IO_file_write@@GLIBC_2.17
gethostent_r@@GLIBC_2.17
_IO_proc_close@@GLIBC_2.17
getservent_r@@GLIBC_2.17
_IO_file_fopen@@GLIBC_2.17
getprotobyname_r@@GLIBC_2.17
sched_getaffinity@@GLIBC_2.17
__tls_get_addr@@GLIBC_2.17
getprotobynumber_r@@GLIBC_2.17
_IO_file_underflow@@GLIBC_2.17
_dl_argv@@GLIBC_PRIVATE
pthread_attr_init@@GLIBC_2.17
_IO_fsetpos64@@GLIBC_2.17

x86_64:

[conda@96dbbda9d9fa ~]$ ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[conda@96dbbda9d9fa ~]$ ld --version
GNU ld version 2.27-44.base.el7_9.1
Copyright (C) 2016 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

strings /usr/lib64/libc.so.6 | grep GLIBC
GLIBC_2.2.5
GLIBC_2.2.6
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.3.4
GLIBC_2.4
GLIBC_2.5
GLIBC_2.6
GLIBC_2.7
GLIBC_2.8
GLIBC_2.9
GLIBC_2.10
GLIBC_2.11
GLIBC_2.12
GLIBC_2.13
GLIBC_2.14
GLIBC_2.15
GLIBC_2.16
GLIBC_2.17
GLIBC_PRIVATE
___sys_nerr_GLIBC_2_3
___sys_nerr_GLIBC_2_4
___sys_nerr_GLIBC_2_1
___sys_errlist_GLIBC_2_1
___sys_errlist_GLIBC_2_4
___sys_errlist_GLIBC_2_3
_IO_file_seekoff@@GLIBC_2.2.5
getservent_r@@GLIBC_2.2.5
_sys_siglist@@GLIBC_2.3.3
_dl_starting_up@@GLIBC_PRIVATE
tmpfile@@GLIBC_2.2.5
sched_getaffinity@GLIBC_2.3.3
pthread_cond_wait@GLIBC_2.2.5
regexec@GLIBC_2.2.5
memcpy@@GLIBC_2.14
_IO_file_init@@GLIBC_2.2.5
realpath@GLIBC_2.2.5
realpath@@GLIBC_2.3
__ctype32_tolower@GLIBC_2.2.5
gethostbyname2_r@@GLIBC_2.2.5
getaliasbyname_r@@GLIBC_2.2.5
_IO_file_overflow@@GLIBC_2.2.5
pthread_cond_wait@@GLIBC_2.3.2
getprotoent_r@@GLIBC_2.2.5
regexec@@GLIBC_2.3.4
_rtld_global@@GLIBC_PRIVATE
getgrgid_r@@GLIBC_2.2.5
pthread_cond_init@GLIBC_2.2.5
_IO_fsetpos@@GLIBC_2.2.5
GLIBC_2.8
_IO_proc_close@@GLIBC_2.2.5
_sys_nerr@@GLIBC_2.12
pthread_cond_timedwait@@GLIBC_2.3.2
__libc_enable_secure@@GLIBC_PRIVATE
pclose@@GLIBC_2.2.5
getaliasent_r@@GLIBC_2.2.5
__tls_get_addr@@GLIBC_2.3
_sys_nerr@GLIBC_2.4
sys_sigabbrev@GLIBC_2.2.5
pthread_cond_broadcast@GLIBC_2.2.5
_IO_proc_open@@GLIBC_2.2.5
posix_spawn@@GLIBC_2.15
getpwnam_r@@GLIBC_2.2.5
_rtld_global_ro@@GLIBC_PRIVATE
getrpcbyname_r@@GLIBC_2.2.5
pthread_cond_broadcast@@GLIBC_2.3.2
nftw64@GLIBC_2.2.5
_sys_siglist@GLIBC_2.2.5
_sys_nerr@GLIBC_2.3
_IO_fsetpos64@@GLIBC_2.2.5
__ctype_tolower@GLIBC_2.2.5
__ctype_toupper@GLIBC_2.2.5
_IO_file_underflow@@GLIBC_2.2.5
localeconv@@GLIBC_2.2.5
pthread_cond_timedwait@GLIBC_2.2.5
_IO_file_fopen@@GLIBC_2.2.5
__ctype_b@GLIBC_2.2.5
GLIBC_2.5
_IO_file_finish@@GLIBC_2.2.5
GLIBC_2.9
getgrnam_r@@GLIBC_2.2.5
GLIBC_2.7
GLIBC_2.6
getservbyname_r@@GLIBC_2.2.5
__ctype32_b@GLIBC_2.2.5
_res@GLIBC_2.2.5
__secure_getenv@GLIBC_2.2.5
getnetent_r@@GLIBC_2.2.5
nftw@GLIBC_2.2.5
getpwuid_r@@GLIBC_2.2.5
_IO_fdopen@@GLIBC_2.2.5
posix_spawn@GLIBC_2.2.5
gethostbyaddr_r@@GLIBC_2.2.5
_IO_file_xsputn@@GLIBC_2.2.5
pthread_attr_init@@GLIBC_2.2.5
getrpcbynumber_r@@GLIBC_2.2.5
_IO_fopen@@GLIBC_2.2.5
getnetbyname_r@@GLIBC_2.2.5
memcpy@GLIBC_2.2.5
pthread_cond_signal@@GLIBC_2.3.2
pthread_cond_signal@GLIBC_2.2.5
sys_sigabbrev@@GLIBC_2.3.3
_IO_fgetpos@@GLIBC_2.2.5
posix_spawnp@@GLIBC_2.15
_IO_file_setbuf@@GLIBC_2.2.5
getservbyport_r@@GLIBC_2.2.5
_sys_errlist@GLIBC_2.3
_IO_file_attach@@GLIBC_2.2.5
gethostent_r@@GLIBC_2.2.5
_sys_errlist@GLIBC_2.4
_sys_nerr@GLIBC_2.2.5
nftw64@@GLIBC_2.3.3
_IO_fgetpos64@@GLIBC_2.2.5
__res_maybe_init@GLIBC_PRIVATE
fnmatch@@GLIBC_2.2.5
h_errno@GLIBC_PRIVATE
_sys_errlist@@GLIBC_2.12
_IO_fclose@@GLIBC_2.2.5
gethostbyname_r@@GLIBC_2.2.5
_IO_do_write@@GLIBC_2.2.5
getspent_r@@GLIBC_2.2.5
getpwent_r@@GLIBC_2.2.5
pthread_cond_destroy@GLIBC_2.2.5
sched_setaffinity@GLIBC_2.3.3
getspnam_r@@GLIBC_2.2.5
sched_setaffinity@@GLIBC_2.3.4
_IO_popen@@GLIBC_2.2.5
getgrent_r@@GLIBC_2.2.5
_IO_file_sync@@GLIBC_2.2.5
GLIBC_2.11
GLIBC_2.16
GLIBC_2.10
pthread_cond_init@@GLIBC_2.3.2
GLIBC_2.17
getprotobyname_r@@GLIBC_2.2.5
GLIBC_2.13
getnetbyaddr_r@@GLIBC_2.2.5
_sys_errlist@GLIBC_2.2.5
_IO_file_write@@GLIBC_2.2.5
pthread_cond_destroy@@GLIBC_2.3.2
nftw@@GLIBC_2.3.3
GLIBC_2.2.6
fopencookie@@GLIBC_2.2.5
__ctype32_toupper@GLIBC_2.2.5
sched_getaffinity@@GLIBC_2.3.4
_dl_argv@@GLIBC_PRIVATE
posix_spawnp@GLIBC_2.2.5
getrpcent_r@@GLIBC_2.2.5
getprotobynumber_r@@GLIBC_2.2.5
_IO_file_close_it@@GLIBC_2.2.5
_obstack@GLIBC_2.2.5

@corneliusroemer
Copy link
Member Author

corneliusroemer commented Mar 5, 2024 •
edited
Loading

Thanks @martin-g! That's quite an old version and current Rust versions require at least glibc 2.17 Rust announcement
which should be fine. However for some reason sometimes glibc 2.18 is in fact required, see blog post

Trying to see what the difference between aarch and x86 is, I noticed that aarch has the following entry which x86 doesn't:

aarch:

$ strings /usr/lib64/libc.so.6 | grep GLIBC_2.18
GLIBC_2.18

x86:

$ strings /usr/lib64/libc.so.6 | grep GLIBC_2.18

Could this be related to it? What do you suggest we do in this case? Switch off tests and wait for builders to use more modern centos? We could package the musl build, but that's got terrible performance for multi-threading, an important feature of Nextclade.

See Rust issue: rust-lang/rust#57497

Centos7 will reach end of life in June 2024: https://www.redhat.com/en/events/webinar/centos-linux-reaching-its-end-life-now-what

@jmarshall
Copy link
Member

The glibc version is controlled by the sysroot_linux-ARCH package pulled in by conda-build. For us, it is specified by conda-forge-pinning's conda_build_config.yaml as described here and here:

c_stdlib_version:              # [unix]
  - 2.12                       # [linux and x86_64]
  - 2.17                       # [linux and not x86_64]

(Your checking of /usr/lib64/libc.so.6 versions above appears to be checking the OS-installed glibc rather than a conda-supplied one?!)

It is indeed annoying that these are so old. I had to patch htslib to add linux-aarch64 support to work around the ancient 2.17 to avoid samtools/htscodecs#88 (comment) and now I finally understand why conda builds choose this ancient version!

The first “here” link in my first paragraph shows how to change the default by adding your own conda_build_config.yaml to your recipe. This may or may not have undesired side-effects, and may not be relevant for your binary-copying recipe. I used a different approach for htslib before I found that documentation.

Updating these is conda-forge/conda-forge.github.io#1941 but would seem to be a long way off — see the discussion there and e.g. conda-forge/ctng-compilers-feedstock#109 (comment).

@corneliusroemer
Copy link
Member Author

corneliusroemer commented Mar 8, 2024 •
edited
Loading

Thanks @jmarshall, that's very insightful! So the glibc versions do seem to be different after all - which may or may not explain why I see failure only in aarch64 and not in arm64.

I'll try to change the build config to see if that fixes it.

Update: Unfortunately setting sysroot_linux-aarch64 2.28.* in the hope of getting a builder with __glibc >=2.28 doesn't work as there doesn't seem to be a builder available with such a minimum glibc version. Shame.

21:11:23 BIOCONDA INFO (OUT) RuntimeError: Solver could not find solution.Mamba failed to solve:
21:11:23 BIOCONDA INFO (OUT)  - __glibc >=2.28,<3.0.a0
21:11:23 BIOCONDA INFO (OUT)  - sysroot_linux-aarch64 2.28.*

https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/110243/workflows/238b4621-96e8-4725-b174-553c497413ba/jobs/181068

Other recipes that repackage binaries and mention glibc in the context of having build/test issues often resort to disabling tests. I guess that's the workaround for now:

@martin-g martin-g mentioned this issue Mar 13, 2024
Closed
@martin-g
Copy link
Contributor

#46341 is a similar issue.
Is it correct to say that the tests try to use the OS-installed glibc ? Is there a way to preload the sysroot_linux-ARCH one ?

This was referenced Mar 20, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
aarch64 Related to adding linux-aarch64 support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jmarshall @martin-g @corneliusroemer