PETSc configure and geosx linking

[andrea-franceschini]

With the current CMakeLists.txt, I'm not able to compile PETSc.
I have an error during the PETSc configure:

Warning! Incorrect argument type specified: -with-mpi=0 --download-ptscotch --with-superlu_dist-dir=/home/franc90/code/geosx/thirdPartyLibs/install-default-release/superlu_dist --with-parmetis-dir=/home/franc90/code/geosx/thirdPartyLibs/install-default-release/parmetis --with-hypre-dir=/home/franc90/code/geosx/thirdPartyLibs/install-default-release/hypre --with-blaslapack-dir= - expecting type Bool.

Then, everything compiles but, when linking geosx, it looks for PCHYPRESetType and it is missing (I think it should be in the PETSc lib). A ldd of the PETSc lib doesn't show any hypre contents.

According to the error from PETSc configure, the wrong part of the code should be:
https://github.com/GEOSX/thirdPartyLibs/blob/089a2a5bbeda91a2b3bd4f30c389d8c21959b0a5/CMakeLists.txt#L798

I'm using Ubuntu 18.04, with cmake 3.10.2, gcc/g++/gfortran 7.4.0 and openmpi 2.1.1.

[andrea-franceschini]

After #60, I'm able to compile the tpl, but linking GEOSX is still a problem. The error that I have is:
libpetsc.so: undefined reference to `SCOTCH_ParMETIS_V3_NodeND'

[TotoGaz]

Hi, There is probably something wrong with the CMakeLists.txt and the ENABLE_MPI option. You may have compiled PETSc without ptscotch; have a look in your TPL/petsc/lib (or so…) folder and check if you more than just libpetsc.so*. Th. From: Andrea Franceschini <[email protected]> Reply-To: GEOSX/thirdPartyLibs <[email protected]> Date: Tuesday, September 17, 2019 at 1:41 PM To: GEOSX/thirdPartyLibs <[email protected]> Cc: Thomas GAZZOLA <[email protected]>, Assign <[email protected]> Subject: Re: [GEOSX/thirdPartyLibs] PETSc configure and geosx linking (#63) After #60<#60>, I'm able to compile the tpl, but linking GEOSX is still a problem. The error that I have is: libpetsc.so: undefined reference to `SCOTCH_ParMETIS_V3_NodeND' — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#63>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALV4DD3GDIPKRW3VZJ6G4ATQKE6FLANCNFSM4IXC5Q4Q>.

[andrea-franceschini]

Everything is set up according to the default options and MPI should be enabled.
In the installation folder I have:

libpetsc.so
libpetsc.so.3.10
libpetsc.so.3.10.2
libptesmumps.a
libptscotch.a
libptscotcherr.a
libptscotcherrexit.a
libptscotchparmetis.a
libscalapack.a
libscotch.a
libscotcherr.a
libscotcherrexit.a
libscotchmetis.a
petsc
pkgconfig

[rrsettgast]

have you tried a newer cmake?

[andrea-franceschini]

Not yet! I'm trying to use a local installation, but the module folder conflicts with the system one (/usr/share/cmake-3.10/Modules) ...

[rrsettgast]

Can you confirm that it doesn't work on another system?

[TotoGaz]

Can you find SCOTCH_ParMETIS_V3_NodeND (e.g. with nm) in these libraries ? Th. From: Andrea Franceschini <[email protected]> Reply-To: GEOSX/thirdPartyLibs <[email protected]> Date: Tuesday, September 17, 2019 at 4:05 PM To: GEOSX/thirdPartyLibs <[email protected]> Cc: Thomas GAZZOLA <[email protected]>, Assign <[email protected]> Subject: Re: [GEOSX/thirdPartyLibs] PETSc configure and geosx linking (#63) Everything is set up according to the default options and MPI should be enabled. In the installation folder I have: libpetsc.so libpetsc.so.3.10 libpetsc.so.3.10.2 libptesmumps.a libptscotch.a libptscotcherr.a libptscotcherrexit.a libptscotchparmetis.a libscalapack.a libscotch.a libscotcherr.a libscotcherrexit.a libscotchmetis.a petsc pkgconfig — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#63>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALV4DD4HAAKFN2VYO2GSTSDQKFPDNANCNFSM4IXC5Q4Q>.

[andrea-franceschini]

Can you confirm that it doesn't work on another system?

I don't have another system ... Do you mean something different from Ubuntu 18.04?

[andrea-franceschini]

Yes, it's available in libptscotchparmetis.a only.

[TotoGaz]

For info, I compiled TPL + GEOSX entirely today. I had issues with the ENABLE_MPI (I believe), but in the end, I managed to do it. Th. From: Randolph Settgast <[email protected]> Reply-To: GEOSX/thirdPartyLibs <[email protected]> Date: Tuesday, September 17, 2019 at 5:00 PM To: GEOSX/thirdPartyLibs <[email protected]> Cc: Thomas GAZZOLA <[email protected]>, Assign <[email protected]> Subject: Re: [GEOSX/thirdPartyLibs] PETSc configure and geosx linking (#63) Can you confirm that it doesn't work on another system? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#63>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALV4DDYSMPSVBZ4QFWOQBM3QKFVQRANCNFSM4IXC5Q4Q>.

[andrea-franceschini]

Today I compiled everything from scratch (both TPL and GEOSX) and I always have the same error:
libpetsc.so: undefined reference to `SCOTCH_ParMETIS_V3_NodeND'

[AntoineMazuyer]

btw, I also have a similar problem :

/usr/bin/ld: gk_cur_jbufs: TLS definition in lib/libgeosx_core.so section .tdata mismatches non-TLS definition in /usr/lib/libmetis.so section .data
/usr/bin/ld: /usr/lib/libmetis.so: error adding symbols: bad value
clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/geosx.dir/build.make:143: bin/geosx] Error 1
make[1]: *** [CMakeFiles/Makefile2:358: CMakeFiles/geosx.dir/all] Error 2
make: *** [Makefile:141: all] Error 2

When compiling geosx :/

Maybe it is related to the recent changes in the tpls ?

[TotoGaz]

Do you have the linking command line (maybe VERBOSE=1 will help)? Did you recompile your petsc or superlu recently? Is it possible that it went wrong ? Do you have a libmetis in your TPL ? Th. From: Antoine Mazuyer <[email protected]> Reply-To: GEOSX/thirdPartyLibs <[email protected]> Date: Wednesday, September 18, 2019 at 10:30 AM To: GEOSX/thirdPartyLibs <[email protected]> Cc: Thomas GAZZOLA <[email protected]>, Assign <[email protected]> Subject: Re: [GEOSX/thirdPartyLibs] PETSc configure and geosx linking (#63) btw, I also have a similar problem : /usr/bin/ld: gk_cur_jbufs: TLS definition in lib/libgeosx_core.so section .tdata mismatches non-TLS definition in /usr/lib/libmetis.so section .data /usr/bin/ld: /usr/lib/libmetis.so: error adding symbols: bad value clang-8: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: *** [CMakeFiles/geosx.dir/build.make:143: bin/geosx] Error 1 make[1]: *** [CMakeFiles/Makefile2:358: CMakeFiles/geosx.dir/all] Error 2 make: *** [Makefile:141: all] Error 2 When compiling geosx :/ — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#63>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALV4DD2ZCPVGYRE6VRXBMCTQKJQT3ANCNFSM4IXC5Q4Q>.

[andrea-franceschini]

Having a closer look to the PETSc library problem, I notice that, in the cmake inital configuration of GEOSX, the following variable is set:
Petsc_LIBRARIES = .../code/geosx/thirdPartyLibs/install-default-release/petsc/lib/libpetsc.so
and the libptscotchparmetis.a is completely missing.
Can this be the problem?

[francoishamon]

@AF1990 @TotoGaz I had linking problems earlier this week, but I managed to compile the TPLs and GEOSX on my machine without any problem after pulling the recent changes this morning. My system is Ubuntu 18.04, gcc 7.4.0, openmpi 2.1.1.

[andrea-franceschini]

I really don't understand where the difference is 😢
I'm always facing the scotch error ...

[francoishamon]

@AF1990 Strange... and we are using the same version of CMake (3.10.2)

I don't know if it helps for your specific problem, but the only thing that I had to change on my system to solve an issue with Scotch and compile the TPLs was:
sudo apt-get install flex bison
and then from there everything went smoothly.

[andrea-franceschini]

I already have both of them ...

[AntoineMazuyer]

Do you have the linking command line (maybe VERBOSE=1 will help)? Did you recompile your petsc or superlu recently? Is it possible that it went wrong ? Do you have a libmetis in your TPL ?

I think the problem comes from metis which is already integrated in PETSc so there is two concurrent versions and the linker is lost somehow.

[andrea-franceschini]

Digging into the problem, it turns out that the already present scotch library on my system seems to be the main issue. It is compiled without the SCOTCH_METIS_PREFIX that PETSc requires. When PETSc links to create libpetsc.so, it uses the system library (why???), leaving undefined some functions, like SCOTCH_ParMETIS_V3_NodeND, that are present without the SCOTCH_ prefix, i.e., ParMETIS_V3_NodeND.
Any idea on how to solve the problem (the possibility to remove scotch from the system is not one!)?

[andrea-franceschini]

Josh, I added you because I know you are working with PETSc now. Maybe you are interested in this problem.