Parallel make install is missing dependencies #405

mweinelt · 2023-03-15T17:04:09Z

For nixpkgs we are converting our install hook to parallelize installation, which yields great speedups with e.g., openssl.

For solanum it was noticed, that it does not play well with for example make -j6 install. It consistently fails like this:

solanum> /nix/store/1qasgqvab0xh2jcy00x9b1zh39dw7m8f-binutils-2.40/bin/ld: cannot find ./.libs/libircd.so: No such file or directory
solanum> collect2: error: ld returned 1 exit status
solanum> make[4]: *** [Makefile:633: solanum] Error 1
solanum> make[4]: *** Waiting for unfinished jobs....

Solanum is one of 12 packages, for which we had to disable parallel install. Would be amzing if this dependency could get set up.

The text was updated successfully, but these errors were encountered:

aaronmdjones · 2023-03-15T17:43:39Z

I'm having trouble reproducing this.

Specifically; the following has worked for 10 consecutive times:

$ rm -rf /home/aaron/ircd/{bin,lib,modules}/ && \
    ./configure --prefix=/home/aaron/ircd/ && \
    make -j install && \
    make distclean && \
    echo ok

Furthermore, if I'm reading the output of your error log properly, the problem is that it's trying to build ircd/solanum before building ircd/libircd.la (from ircd/Makefile.am). However, solanum_LDADD has libircd.la in it, so automake ought not to be generating a Makefile that attempts to do so, which I'm preliminarily confirming by looking at ircd/Makefile.in and observing solanum_DEPENDENCIES = libircd.la

Can you provide more information?

trofi · 2023-03-15T18:59:17Z

Initially we observed the install failure on make install -j2. Here is the full build log: https://hydra.nixos.org/log/6d9j7mvknsyvkhf38v238m4g18ra30cx-solanum-unstable-2022-07-12.drv

A few suggestions to trigger it effectively:

use --shuffle option for GNU make 4.4 or later to reorder prerequisite execution
along with high parallelism also try to use low parallelism level, like -j2. that gives more chance to execute only subset of prerequisites
make sure your /usr/lib/libircd.so (or other system pah) is not present. Otherwise you would not be able to reproduce the failure as relinking will accidentally happen against the system library.

For me using the above against solanum from e593329 commit (latest main) I get mainly install failures (and less frequently successes):

# a bit too specific to my setup
$ while :; do nix build --impure --expr 'with import ~/n {}; solanum.overrideAttrs (oa: { src = builtins.fetchGit ./.; })' --cores 2 --rebuild; echo again; done

error: builder for '/nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv' failed with exit code 2;
       last 10 log lines:
       > more information, such as the ld(1) and ld.so(8) manual pages.
       > ----------------------------------------------------------------------
       > make[4]: Leaving directory '/build/source/ircd'
       > make[3]: *** [Makefile:831: install-am] Error 2 shuffle=2912249958
       > make[3]: Leaving directory '/build/source/ircd'
       > make[2]: *** [Makefile:824: install] Error 2 shuffle=2912249958
       > make[2]: Leaving directory '/build/source/ircd'
       > make[1]: *** [Makefile:474: install-recursive] Error 1 shuffle=2912249958
       > make[1]: Leaving directory '/build/source'
       > make: *** [Makefile:776: install] Error 2 shuffle=2912249958
       For full logs, run 'nix log /nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv'.
error:
again
error: builder for '/nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv' failed with exit code 2;
       last 10 log lines:
       > more information, such as the ld(1) and ld.so(8) manual pages.
       > ----------------------------------------------------------------------
       > make[4]: Leaving directory '/build/source/ircd'
       > make[3]: *** [Makefile:831: install-am] Error 2 shuffle=2889377224
       > make[3]: Leaving directory '/build/source/ircd'
       > make[2]: *** [Makefile:824: install] Error 2 shuffle=2889377224
       > make[2]: Leaving directory '/build/source/ircd'
       > make[1]: *** [Makefile:474: install-recursive] Error 1 shuffle=2889377224
       > make[1]: Leaving directory '/build/source'
       > make: *** [Makefile:776: install] Error 2 shuffle=2889377224
       For full logs, run 'nix log /nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv'.
again
error: builder for '/nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv' failed with exit code 2;
       last 10 log lines:
       > more information, such as the ld(1) and ld.so(8) manual pages.
       > ----------------------------------------------------------------------
       > make[4]: Leaving directory '/build/source/ircd'
       > make[3]: *** [Makefile:831: install-am] Error 2 shuffle=2869706743
       > make[3]: Leaving directory '/build/source/ircd'
       > make[2]: *** [Makefile:824: install] Error 2 shuffle=2869706743
       > make[2]: Leaving directory '/build/source/ircd'
       > make[1]: *** [Makefile:474: install-recursive] Error 1 shuffle=2869706743
       > make[1]: Leaving directory '/build/source'
       > make: *** [Makefile:776: install] Error 2 shuffle=2869706743
       For full logs, run 'nix log /nix/store/7gy1dvmnp70ymy4nlv16h2fc6xljv2nj-solanum-unstable-2022-07-12.drv'.
error:

Thus I should be able to verify the fix if you will have a hard time triggering it on your machine.

Looking at the failure specifics (against current main) this is the full install log:

installing
install flags: -j2 SHELL=/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash install
make  install-recursive
make[1]: Entering directory '/build/source'
Making install in librb
make[2]: Entering directory '/build/source/librb'
make  install-recursive
make[3]: Entering directory '/build/source/librb'
Making install in src
make[4]: Entering directory '/build/source/librb/src'
make  install-am
make[5]: Entering directory '/build/source/librb/src'
make[6]: Entering directory '/build/source/librb/src'
make[6]: Nothing to be done for 'install-data-am'.
 /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/mkdir -p '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib'
 /nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash ../libtool --silent   --mode=install /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c   librb.la '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib'
make[6]: Leaving directory '/build/source/librb/src'
make[5]: Leaving directory '/build/source/librb/src'
make[4]: Leaving directory '/build/source/librb/src'
make[4]: Entering directory '/build/source/librb'
make[5]: Entering directory '/build/source/librb'
make[5]: Nothing to be done for 'install-exec-am'.
 /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/mkdir -p '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib/pkgconfig'
 /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c -m 644 librb.pc '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib/pkgconfig'
make[5]: Leaving directory '/build/source/librb'
make[4]: Leaving directory '/build/source/librb'
make[3]: Leaving directory '/build/source/librb'
make[2]: Leaving directory '/build/source/librb'
Making install in ircd
make[2]: Entering directory '/build/source/ircd'
/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash version.c.SH
Extracting solanum/src/version.c...
/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/cp version.c version.c.last
touch version.c.SH
make  install-am
make[3]: Entering directory '/build/source/ircd'
/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash version.c.SH
Extracting solanum/src/version.c...
/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/cp version.c version.c.last
touch version.c.SH
  CC       version.lo
  CCLD     libircd.la
  CCLD     solanum
make[4]: Entering directory '/build/source/ircd'
/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash version.c.SH
Extracting solanum/src/version.c...
/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/cp version.c version.c.last
touch version.c.SH
  CC       version.lo
  CCLD     libircd.la
 /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/mkdir -p '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib'
 /nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash ../libtool   --mode=install /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c   libircd.la '/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib'
  CCLD     solanum
libtool: warning: relinking 'libircd.la'
libtool: install: (cd /build/source/ircd; /nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin/bash "/build/source/libtool"  --silent --tag CC --mode=relink gcc -g -O2 -avoid-version -no-undefined -Wl,-rpath=/nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib -o libircd.la -rpath /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib authproc.lo bandbi.lo cache.lo capability.lo channel.lo chmode.lo class.lo client.lo dns.lo extban.lo getopt.lo hash.lo hook.lo hostmask.lo ircd.lo ircd_parser.lo ircd_lexer.lo ircd_signal.lo listener.lo logger.lo match.lo modules.lo monitor.lo msgbuf.lo newconf.lo operhash.lo packet.lo parse.lo privilege.lo ratelimit.lo reject.lo restart.lo s_conf.lo s_newconf.lo s_serv.lo s_user.lo scache.lo send.lo snomask.lo sslproc.lo substitution.lo supported.lo tgchange.lo version.lo whowas.lo wsproc.lo -lltdl -L../librb/src -lrb )
/nix/store/r7yp62k0cqhm59nrvn8wn1xsz0kl3sgv-binutils-2.40/bin/ld: cannot find ./.libs/libircd.so: No such file or directory
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:633: solanum] Error 1 shuffle=2854295066
make[4]: *** Waiting for unfinished jobs....
libtool: install: /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c .libs/libircd.soT /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib/libircd.so
libtool: install: /nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin/install -c .libs/libircd.lai /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib/libircd.la
libtool: finish: PATH="/nix/store/xwd7pd4fd4zl28m50jczvf7k5rynsi71-autoconf-2.69/bin:/nix/store/q4pv8wl5p7wwx34ggcm6fv8li7qr8wkb-automake-1.16.5/bin:/nix/store/wp9hly5270bxy5ldnv4n6rrb7yc20zxb-gettext-0.21/bin:/nix/store/m583s17s13n9d81qnzd37c0d37li2z7p-libtool-2.4.7/bin:/nix/store/yk6g0lc7sgqghf7dxv654kz4sy8ab7iv-gnum4-1.4.19/bin:/nix/store/hx73nmjw84hxk5jnf081ncp1jybish8w-file-5.44/bin:/nix/store/x3w26b833bhmvxd3wyzhfva94dvllvnm-bison-3.8.2/bin:/nix/store/4kqa6bdhnnllpglmdhy0n0zw1im4lair-flex-2.6.4/bin:/nix/store/k66cg7smlvlc4dylny6fsp8cw2c9g52g-pkg-config-wrapper-0.29.2/bin:/nix/store/i8sfwz7pmxar03k98k5vwqkv0w00inmy-util-linux-2.38.1-bin/bin:/nix/store/fc0ypma9gi7r85qvn22w5x5934br6a3r-patchelf-0.17.2/bin:/nix/store/69iqwrsr5dcfi2fsqf5w3hmx15pmfaqk-gcc-wrapper-13.0.0/bin:/nix/store/v626s1w0s6hdnjxlaig37q9wzqm63ah9-gcc-13.0.0/bin:/nix/store/w7kz70f1dw8waf803lddv3isb9bhjpfk-glibc-2.35-224-bin/bin:/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin:/nix/store/vrk8hj570x5nm2j65pc3j634ygcc63hx-binutils-wrapper-2.40/bin:/nix/store/r7yp62k0cqhm59nrvn8wn1xsz0kl3sgv-binutils-2.40/bin:/nix/store/173mgp8qnpzs5bznhh69kdyy4mlk62bn-openssl-3.0.8-bin/bin:/nix/store/35nb54v62wnmh2ix4zp84ycnmgvavjm8-sqlite-3.41.1-bin/bin:/nix/store/0vq35r0amg9dmdr0vn0kbn6pslgrfyvi-coreutils-9.1/bin:/nix/store/g5pbad773hzx4j54i5dn22bkjzzigb6h-findutils-4.9.0/bin:/nix/store/285vm1zn78dy87adny0z0sbwkwbyrwvi-diffutils-3.9/bin:/nix/store/7xa8bc3ipyz1y9jl1p6iyjb30vxfk1zq-gnused-4.9/bin:/nix/store/2iaqkjcdma1bw19g0dg1zybc9w2jq413-gnugrep-3.7/bin:/nix/store/1l3vkqzp7bppjhh06w7lgfp1662nk26h-gawk-5.2.1/bin:/nix/store/cs2m8536x9hrbxd4yzg2haspy368757k-gnutar-1.34/bin:/nix/store/iykzhqbr570zisqcvdyq6s3hh42zns4f-gzip-1.12/bin:/nix/store/knz3cacv109gzvf75mgag6bvjqvk9ra2-bzip2-1.0.8-bin/bin:/nix/store/2p4ri16y7rjyir3l9qdxic5mdnshix5b-gnumake-4.4.1/bin:/nix/store/faf33ca369ww772hz9x0an7zx7v8bgyw-bash-5.2-p15/bin:/nix/store/b77k73j5is2s7c5af87rv7p06765bhwi-patch-2.7.6/bin:/nix/store/nicb37s3z16116kqyaq959i8gs0pvmi6-xz-5.4.1-bin/bin:/nix/store/pqihdci4c2d8g1ryihk8wh654pnwxfvx-file-5.44/bin:/sbin" ldconfig -n /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib
----------------------------------------------------------------------
Libraries have been installed in:
   /nix/store/8i3r68hdfg028h0sr5mrb2lkqc5j1k0s-solanum-unstable-2022-07-12/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
make[4]: Leaving directory '/build/source/ircd'
make[3]: *** [Makefile:831: install-am] Error 2 shuffle=2854295066
make[3]: Leaving directory '/build/source/ircd'
make[2]: *** [Makefile:824: install] Error 2 shuffle=2854295066
make[2]: Leaving directory '/build/source/ircd'
make[1]: *** [Makefile:474: install-recursive] Error 1 shuffle=2854295066
make[1]: Leaving directory '/build/source'
make: *** [Makefile:776: install] Error 2 shuffle=2854295066

I find it suspicions that libircd.la is attempted to be built multiple times (and CC is executed at all during the install). I wonder if it gets deleted as a temporary eventually. I'll add V=1 to get more detailed logs.

trofi · 2023-03-15T19:07:05Z

solanum-unstable-V-1.log.txt is the GNUMAKEFLAGS=V=1 make install -j2 with a similar failure. Could it be that version.c.SH rebuild causes removal of .libs/libircd.so?

I find this multiple removals suspicious:

$ cat /tmp/solanum-unstable-V-1.log.txt | grep -P 'rm.*libircd.so'

libtool: link:  gcc -shared  -fPIC -DPIC  .libs/extb_usermode.o   -Wl,-rpath -Wl,/build/source/ircd/.libs -Wl,-rpath -Wl,/nix/store/xgmkk41j25qi1nj7gy4bxrgc6b9kdz5g-libtool-2.4.7-lib/lib -Wl,-rpath -Wl,/build/source/librb/src/.libs -Wl,-rpath -Wl,/nix/store/xvf1sawg92x6cp8plw34ica3l5y88qvb-solanum-unstable-2022-07-12/lib -Wl,-rpath -Wl,/nix/store/xgmkk41j25qi1nj7gy4bxrgc6b9kdz5g-libtool-2.4.7-lib/lib -L/build/source/librb/src/.libs ../ircd/.libs/libircd.so -L/nix/store/xgmkk41j25qi1nj7gy4bxrgc6b9kdz5g-libtool-2.4.7-lib/lib -L../librb/src -L/nix/store/v3y5ds8z0y128gspghnmkrxxdh3xc4j8-openssl-3.0.8-dev/lib /nix/store/xgmkk41j25qi1nj7gy4bxrgc6b9kdz5g-libtool-2.4.7-lib/lib/libltdl.so /build/source/librb/src/.libs/librb.so -lssl -lcrypto  -g -O2 -Wl,-rpath=/nix/store/xvf1sawg92x6cp8plw34ica3l5y88qvb-solanum-unstable-2022-07-12/lib   -Wl,-soname -Wl,extb_usermode.so -Wl,-version-script -Wl,.libs/extb_usermode.ver -o .libs/extb_usermode.so
libtool: link: rm -fr  .libs/libircd.la .libs/libircd.lai .libs/libircd.so
libtool: link: rm -fr  .libs/libircd.la .libs/libircd.lai .libs/libircd.so
libtool: link: rm -fr  .libs/libircd.la .libs/libircd.lai .libs/libircd.so

trofi · 2023-03-15T19:14:03Z

I think it gets rebuild indefinitely because of touch version.c.SH:

version.c: version.c.SH ../CREDITS ../include/patchlevel.h ../include/serno.h
        $(SHELL) version.c.SH
        $(CP) version.c version.c.last
        touch version.c.SH

That touch version.c.SH makes this rule fire forever. Should it be touch version.c instead?

trofi · 2023-03-15T19:16:25Z

On top of that removal of the source in the middle of install looks fishy as well:

ircd/Makefile.am-install-exec-hook: install-libircdLTLIBRARIES
ircd/Makefile.am:       $(RM) -f version.c

aaronmdjones · 2023-03-15T19:19:36Z

If you remove the touch version.c.SH line entirely (version.c.SH already updates version.c) and remove that $(RM) -f version.c line (but not the line above it expressing an installation dependency), is the problem resolved? I am thus far unable to reproduce this on any parallelism level and my make isn't new enough for the shuffle feature.

trofi · 2023-03-15T21:08:35Z

It was not enough to fix the install. I tried the following change:

--- a/ircd/Makefile.am
+++ b/ircd/Makefile.am
@@ -12,7 +12,6 @@ BUILT_SOURCES = ircd_parser.h version.c
 version.c: version.c.SH ../CREDITS ../include/patchlevel.h ../include/serno.h
        $(SHELL) version.c.SH
        $(CP) version.c version.c.last
-       touch version.c.SH

 libircd_la_SOURCES =                  \
   authproc.c                   \
@@ -71,4 +70,3 @@ solanum_LDFLAGS = $(EXTRA_FLAGS) -dlopen self
 solanum_LDADD = libircd.la -L$(top_srcdir)/librb/src -lrb

 install-exec-hook: install-libircdLTLIBRARIES
-       $(RM) -f version.c

Still fails as:
solanum-unstable-V-1-patched.log.txt

Looking at the log it sill tries to relink libircd.la twice:

libtool: warning: relinking 'libircd.la'
...
libtool: warning: relinking 'libircd.la'
...
ldconfig: File /nix/store/4d5imqc90dqhmi5g22y2mlrfdxxyyxyy-solanum-unstable-2022-07-12/lib/libircd.so is empty, not checked.

trofi · 2023-03-15T21:28:47Z

But if I remove install-exec-hook: install-libircdLTLIBRARIES dependency my rebuild tests survives 10 rebuilds without failures:

--- a/ircd/Makefile.am
+++ b/ircd/Makefile.am
@@ -12,7 +12,6 @@ BUILT_SOURCES = ircd_parser.h version.c
 version.c: version.c.SH ../CREDITS ../include/patchlevel.h ../include/serno.h
        $(SHELL) version.c.SH
        $(CP) version.c version.c.last
-       touch version.c.SH

 libircd_la_SOURCES =                  \
   authproc.c                   \
@@ -69,6 +68,3 @@ bin_PROGRAMS = solanum
 solanum_SOURCES = main.c
 solanum_LDFLAGS = $(EXTRA_FLAGS) -dlopen self
 solanum_LDADD = libircd.la -L$(top_srcdir)/librb/src -lrb
-
-install-exec-hook: install-libircdLTLIBRARIES
-       $(RM) -f version.c

Looking at https://www.gnu.org/software/automake/manual/html_node/Extending-Installation.html I think the desired ordering should already be implied.

aaronmdjones added the bug Something isn't working label Mar 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallel make install is missing dependencies #405

Parallel make install is missing dependencies #405

mweinelt commented Mar 15, 2023

aaronmdjones commented Mar 15, 2023

trofi commented Mar 15, 2023

trofi commented Mar 15, 2023

trofi commented Mar 15, 2023

trofi commented Mar 15, 2023

aaronmdjones commented Mar 15, 2023

trofi commented Mar 15, 2023

trofi commented Mar 15, 2023

Parallel make install is missing dependencies #405

Parallel make install is missing dependencies #405

Comments

mweinelt commented Mar 15, 2023

aaronmdjones commented Mar 15, 2023

trofi commented Mar 15, 2023

trofi commented Mar 15, 2023

trofi commented Mar 15, 2023

trofi commented Mar 15, 2023

aaronmdjones commented Mar 15, 2023

trofi commented Mar 15, 2023

trofi commented Mar 15, 2023