Firedrake tests not passing #3495

Brazovskis · 2024-04-09T12:14:09Z

Brazovskis
Apr 9, 2024

Good afternoon,

I have been facing issues with getting a working installation of Firedrake on an Ubuntu 22.04 machine with Intel Xeon W7 machine.

The installation succeeds with no errors, however, when testing the installation with
pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection"
I get a number of tests failing:

====================================================== short test summary info ======================================================
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic[params1-1-True] - assert 0.9562263984999115 < 1e-09
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic[params2-2-False] - assert 0.583520056710691 < 1e-09
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic[params3-2-True] - firedrake.exceptions.ConvergenceError: Nonlinear solve failed to converge after 25 nonlinear iterations.
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic_linear[params1-1-True] - assert 1.857809736663187 < 5e-06
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic_linear[params2-2-False] - assert 6.649588457517463 < 5e-06
FAILED tests/regression/test_poisson_strong_bcs.py::test_poisson_analytic_linear[params3-2-True] - assert 5.331332146018825 < 5e-06
FAILED tests/regression/test_poisson_strong_bcs_nitsche.py::test_poisson_nitsche[1-True] - assert 3.0350111740039 < 0.001
FAILED tests/regression/test_poisson_strong_bcs_nitsche.py::test_poisson_nitsche[2-False] - assert 230.1658349216025 < 0.001
FAILED tests/regression/test_poisson_strong_bcs_nitsche.py::test_poisson_nitsche[2-True] - assert 5.309348381917987 < 0.001

A similar issue has been discussed in the Firedrake Slack channel before without a definitive solution identified. Following suggestions in said thread, I also provide the following outputs.

Output of ldd $VIRTUAL_ENV/lib/python*/site-packages/mpi4py/MPI.*.so:

linux-vdso.so.1 (0x00007ffccabdd000)
libmpi.so.12 => /home/emils/firedrake/src/petsc/default/lib/libmpi.so.12 (0x00007fef6a600000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007fef6a200000)
libhwloc.so.15 => /home/emils/firedrake/src/petsc/default/lib/libhwloc.so.15 (0x00007fef6ab5b000)
/lib64/ld-linux-x86-64.so.2 (0x00007fef6ad1b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fef6a519000)

Output of PETSc make check:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/emils/firedrake/src/petsc and PETSC_ARCH=default
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
3,5c3,4
<   1 SNES Function norm 3.32845e-06
<   2 SNES Function norm < 1.e-11
< Number of SNES iterations = 2
---
>   1 SNES Function norm 0.0404122
> Number of SNES iterations = 1
/home/emils/firedrake/src/petsc/src/snes/tutorials
Possible problem with ex19 running with SuiteSparse, diffs above
=========================================
2c2
< Number of SNES iterations = 2
---
> Number of SNES iterations = 0
/home/emils/firedrake/src/petsc/src/snes/tutorials
Possible problem with ex19 running with SuperLU_DIST, diffs above
=========================================
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Completed PETSc check examples
Error while running make check
gmake[1]: *** [makefile:131: check] Error 1
make: *** [GNUmakefile:17: check] Error 2

The Firedrake install log:
firedrake-install.log

The PETSc src/petsc/configure.log log:
configure.log

Any help would be greatly appreciated!

Answered by Brazovskis

Apr 15, 2024

Adding a --download-fblaslapack flag to ./configure when installing PETSc seems to have solved the issue. Here's the make check output:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/emils/petsc and PETSC_ARCH=arch-linux-c-opt
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
C/C++ example src/snes/tutorials/ex19 run successfully with SuiteSparse
C/C++ example src/snes/tutorials/ex19 run successfully with SuperLU_DIST
C/C++ ex…

View full answer

connorjward · 2024-04-09T12:36:31Z

connorjward
Apr 9, 2024
Maintainer

The fact that make check is failing suggests that the issue is with PETSc. Could you try compiling PETSc separately? The steps you need to follow are:

Make sure that the current venv isn't activated.
Clone our fork of PETSc
Run (inside the cloned directory)

./configure --with-shared-libraries=1 --download-cmake --download-pastix --download-bison --download-netcdf --download-pnetcdf --download-mpich --download-scalapack --download-mumps --download-suitesparse --download-metis --download-hdf5 --download-ptscotch --download-chaco --download-hypre --with-zlib --download-hwloc --with-c2html=0 --with-debugging=0 --with-fortran-bindings=0 --download-superlu_dist

Run make
Run make check

If this fails to pass then the issue is definitely with PETSc and we can iterate from there.

5 replies

Brazovskis Apr 10, 2024
Author

Getting the same issues when following the steps you provided:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/emils/petsc and PETSC_ARCH=arch-linux-c-opt
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
3,5c3,4
<   1 SNES Function norm 3.32845e-06
<   2 SNES Function norm < 1.e-11
< Number of SNES iterations = 2
---
>   1 SNES Function norm 0.0404122
> Number of SNES iterations = 1
/home/emils/petsc/src/snes/tutorials
Possible problem with ex19 running with SuiteSparse, diffs above
=========================================
2c2
< Number of SNES iterations = 2
---
> Number of SNES iterations = 0
/home/emils/petsc/src/snes/tutorials
Possible problem with ex19 running with SuperLU_DIST, diffs above
=========================================
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Completed PETSc check examples
Error while running make check
gmake[1]: *** [makefile:131: check] Error 1
make: *** [GNUmakefile:17: check] Error 2

connorjward Apr 10, 2024
Maintainer

OK. So the issue is definitely with PETSc. Can you now try running exactly the same steps but with the most recent PETSc (git clone https://gitlab.com/petsc/petsc.git)? If the same errors occur then an issue needs to be submitted to PETSc, if it passes then we need to update our fork (which we updated very recently).

Brazovskis Apr 10, 2024
Author

Yes, seemingly still getting the same errors with the most recent PETSc version:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/emils/petsc and PETSC_ARCH=arch-linux-c-opt
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
3,5c3,4
<   1 SNES Function norm 3.32845e-06
<   2 SNES Function norm < 1.e-11
< Number of SNES iterations = 2
---
>   1 SNES Function norm 0.0404122
> Number of SNES iterations = 1
/home/emils/petsc/src/snes/tutorials
Possible problem with ex19 running with SuiteSparse, diffs above
=========================================
2c2
< Number of SNES iterations = 2
---
> Number of SNES iterations = 0
/home/emils/petsc/src/snes/tutorials
Possible problem with ex19 running with SuperLU_DIST, diffs above
=========================================
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Completed PETSc check examples
Error while running make check
gmake[1]: *** [makefile:131: check] Error 1
make: *** [GNUmakefile:17: check] Error 2

connorjward Apr 10, 2024
Maintainer

OK. Would you be able to submit an issue to PETSc then (and share a link to it here)? Sorry we can't be of further help.

Brazovskis Apr 10, 2024
Author

I've submitted the issue to PETSc here. Thank you for your help!

Brazovskis · 2024-04-15T10:34:55Z

Brazovskis
Apr 15, 2024
Author

Adding a --download-fblaslapack flag to ./configure when installing PETSc seems to have solved the issue. Here's the make check output:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=/home/emils/petsc and PETSC_ARCH=arch-linux-c-opt
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
C/C++ example src/snes/tutorials/ex19 run successfully with SuiteSparse
C/C++ example src/snes/tutorials/ex19 run successfully with SuperLU_DIST
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Completed PETSc check examples

Adding the flag in the firedrake-install script, and then testing the installation with pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection", all the tests seem to pass.

Adding the flag --download-fblaslapack to petsc_options in the firedrake-install file also works.

Thank you for your help!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Firedrake tests not passing #3495

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Firedrake tests not passing #3495

Brazovskis Apr 9, 2024

Replies: 2 comments · 5 replies

connorjward Apr 9, 2024 Maintainer

Brazovskis Apr 10, 2024 Author

connorjward Apr 10, 2024 Maintainer

Brazovskis Apr 10, 2024 Author

connorjward Apr 10, 2024 Maintainer

Brazovskis Apr 10, 2024 Author

Brazovskis Apr 15, 2024 Author

Brazovskis
Apr 9, 2024

Replies: 2 comments 5 replies

connorjward
Apr 9, 2024
Maintainer

Brazovskis Apr 10, 2024
Author

connorjward Apr 10, 2024
Maintainer

Brazovskis Apr 10, 2024
Author

connorjward Apr 10, 2024
Maintainer

Brazovskis Apr 10, 2024
Author

Brazovskis
Apr 15, 2024
Author