Adjoint crashes depending on mesh refinement (and MPI)

[ftosto]

I am running an FFD optimization on a cylindrical annulus. The geometry has been meshed on Gmsh 4.11.1. The simulation is an incompressible RANS with streamwise periodicity on the pressure drop. I am using cylindrical FFD boxes.

I am running everything on 12 processors. The direct simulation runs smoothly; the adjoint simulation, instead, crashes while computing the residuals to check the convergence of the direct problem, without returning a specific error, see below.

By decreasing mesh refinement by a factor of 2, the adjoint simulation still crashes but after the solution has converged, when “Storing computational graph wrt MESH COORDINATES.” The error is the same reported above. However, by reducing the number of processors to 6, the simulation runs successfully. By further coarsening the mesh, the adjoint simulation runs successfully.

I have tried to remove the streamwise periodicity, change the objective function, remove the FFD boxes, and played with other settings but the error persists. It seems to depend on the mesh and, perhaps, on the MPI. A case without MPI on the original mesh has been run too, and it also crashed with the same error.

Desktop (please complete the following information):

• OS: Ubuntu 22.04 WSL on Windows 10
• C++ compiler and version: GCC 11.4
• MPI implementation and version: OpenMPI 4.1.2
• SU2 Version: 8.0.0 (master)

[pcarruscag]

You are probably running out of memory, the adjoint uses 7-10x the memory of the primal, even worse if you use multi grid.