Link errors when running ESMF tests with ESMF_SHARED_LIB_BUILD=OFF on Darwin-gfortranclang

[billsacks]

On my Mac (green, using Darwin-gfortranclang), I'm getting link errors when running ESMF tests with ESMF_SHARED_LIB_BUILD=OFF (things work fine without that setting). The problem seems to come from the recent change to use mpicc instead of mpicxx as a linker: When I run one of the link commands by hand with the now-standard mpicc as a linker:

$ mpicc   -pthread -L/Users/sacks/esmf/esmf1/lib/libg/Darwin.gfortranclang.64.openmpi.default -L/opt/homebrew/Cellar/netcdf/4.9.0/lib -L/opt/homebrew/Cellar/netcdf-fortran/4.6.0/lib -L/opt/homebrew/Cellar/gcc/12.2.0/bin/../lib/gcc/current/gcc/aarch64-apple-darwin21/12/../../../ -L/Users/sacks/esmf/esmf1/lib/libg/Darwin.gfortranclang.64.openmpi.default -L/opt/homebrew/Cellar/netcdf/4.9.0/lib -L/opt/homebrew/Cellar/netcdf-fortran/4.6.0/lib -L/opt/homebrew/Cellar/gcc/12.2.0/bin/../lib/gcc/current/gcc/aarch64-apple-darwin21/12/../../../ -Wl,-rpath,/Users/sacks/esmf/esmf1/lib/libg/Darwin.gfortranclang.64.openmpi.default -Wl,-rpath,/opt/homebrew/Cellar/netcdf/4.9.0/lib -Wl,-rpath,/opt/homebrew/Cellar/netcdf-fortran/4.6.0/lib -Wl,-rpath,/opt/homebrew/Cellar/gcc/12.2.0/bin/../lib/gcc/current/gcc/aarch64-apple-darwin21/12/../../../ -Wl,-rpath,/Users/sacks/esmf/esmf1/lib/libg/Darwin.gfortranclang.64.openmpi.default -Wl,-rpath,/opt/homebrew/Cellar/netcdf/4.9.0/lib -Wl,-rpath,/opt/homebrew/Cellar/netcdf-fortran/4.6.0/lib -Wl,-rpath,/opt/homebrew/Cellar/gcc/12.2.0/bin/../lib/gcc/current/gcc/aarch64-apple-darwin21/12/../../../ -o /Users/sacks/esmf/esmf1/test/testg/Darwin.gfortranclang.64.openmpi.default/ESMC_LogErrUTest  ESMC_LogErrUTest.o -lesmf -lgfortran -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lpioc  -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lpioc

I get a ton of link errors: see attachment.

link_errors.txt

The link works fine if I use mpicxx instead of mpicc.

[theurich]

@billsacks have you tried appending -lstdc++ to the same link line with with mpicc?

[billsacks]

@theurich - I have confirmed that adding -lstdc++ to the end of the link line with mpicc does result in a successful build of the first failing unit test (I haven't tried others, but I'm guessing it would solve the problem for everything). But my (not very experienced) understanding is that it's generally preferred to use a C++ compiler rather than using a C compiler where you explicitly add -lstdc++. If we need to add -lstdc++, is there anything we gain by linking with a C compiler?

[theurich]

@billsacks - thanks for confirming that this works. I also have been thinking about this from a different angle after I made my previous suggestion. Basically my thinking is that since we are setting ESMF_F90LINKERDEFAULT = clang++ in this configuration anyway, we should just do the same for the C side as well for consistency - since potentially the same issues would arise with the C + lstdc++ combo as did for Fortran.
Therefore, I agree, and we should just set ESMF_CLINKERDEFAULT = clang++ in build_config/Darwin.gfortranclang.default/build_rules.mk, like this

############################################################
# Default compiler setting.
#
ESMF_F90DEFAULT         = gfortran
ESMF_F90LINKERDEFAULT   = clang++
ESMF_CXXDEFAULT         = clang++
ESMF_CDEFAULT           = clang
ESMF_CLINKERDEFAULT     = clang++
ESMF_CPPDEFAULT         = clang -E -P -x c

[billsacks]

Thanks, @theurich . I have made that change as well as similar changes for the various MPI compilers in 55a6394. I'll run some testing now to verify that it solves the problem - both with ESMF_SHARED_LIB_BUILD=OFF, and via the test scripts on green. Assuming all looks good, I'll merge that to develop.

This does make me wonder if changes might be needed on other platforms to support the ESMF_SHARED_LIB_BUILD=OFF case. Do you think that's worth testing (e.g., I'm picturing a temporary branch that sets ESMF_SHARED_LIB_BUILD=OFF by default and then having all of our nightly testing point to that branch for a night... but I don't know if that might run into other issues if ESMF_SHARED_LIB_BUILD=OFF isn't a well-supported / well-tested option?)

[theurich]

@billsacks that looks good. Thanks for moving this forward.

I think general testing with ESMF_SHARED_LIB_BUILD=OFF would be interesting, but at the same time I am afraid it might send us down a rabbit hole we really shouldn't be in. I am not sure how generally the OFF option is supported. I certainly do not remember every generally testing it. It is available, and we'd fix it if anybody needed it and found it isn't working. The Darwin issue you are addressing here is a good example for it - it has a clear purpose why it's needed, so it should work. But let's not pro-actively worry about other platforms.

For the Darwin issue, it raises the issue of whether we should include ESMF_SHARED_LIB_BUILD=OFF` in a formal test that is run when Green or Grit do their testing. Of course that adds extra cycles to the testing. It is a bit along the lines of the ticket esmf-org/esmf-test-scripts#86 that talks about testing the various ESMF_TEST_ options. Right now all our testing is done with the same options.

At one point though I guess we just test for the most common combos, and take a more relaxed approach for the rest, i.e. deal with it once we (or a user) notices something is broke. It's just always a question of where that line is drawn.

[billsacks]

I have verified that the changes in 55a6394 solve the issue with ESMF_SHARED_LIB_BUILD=OFF, and tests pass in standard testing as well (i.e., with ESMF_SHARED_LIB_BUILD=ON). I'll merge that to develop.