Ocean EC(wISC)30to60 performance tests fail

[mark-petersen]

In MPAS-Ocean stand-alone testing, the compass tests:

ocean_global_ocean_EC30to60_PHC_performance_test
ocean_global_ocean_ECwISC30to60_PHC_performance_test

fail, or sometimes just hang, when using the gnu optimized compiler on chicoma and chrysalis (and probably others). They pass with gnu debug, intel debug, and intel optimized.

The runs die just before the timestepping begins with the error

$ tail log.ocean.0001.*
----------------------------------------------------------------------
Beginning MPAS-ocean Error Log File for task       1 of     128
    Opened at 2023/04/19 09:15:37
----------------------------------------------------------------------

ERROR: MPAS IO Error: Bad return value from PIO
CRITICAL ERROR: Core init failed for core ocean
Logging complete.  Closing file at 2023/04/19 09:15:37

[xylar]

@mark-petersen, you're seeing this on Chrysalis, too? I hadn't seen that, only on Chicoma and Perlmutter. I ask because Chrysalis is always easier to deal with in terms of queue times so that could be helpful if so.

[mark-petersen]

Correct. Master failed on chrysalis as well, and in the same way as chicoma, and #5610 fixes it.