Investigate performance

[MartinDix]

One day run with n48 configuration shows a stall near the end. E.g. from PET00.ESMF_LogFile

20230802 085707.203 INFO             PET00 UM time in u_model: 1981    9    1    0    0    0
20230802 085710.145 INFO             PET00 UM time in u_model: 1981    9    1    0   20    0
20230802 085710.792 INFO             PET00 UM time in u_model: 1981    9    1    0   40    0
...
20230802 085923.790 INFO             PET00 UM time in u_model: 1981    9    1   22    0    0
20230802 085925.607 INFO             PET00 UM time in u_model: 1981    9    1   22   20    0
20230802 085926.237 INFO             PET00 UM time in u_model: 1981    9    1   22   40    0

so takes about 140 seconds to this point. However then delay of 9 minutes until the next UM step.

20230802 090815.504 INFO             PET00 UM time in u_model: 1981    9    1   23    0    0
20230802 090817.274 INFO             PET00 UM time in u_model: 1981    9    1   23   20    0
20230802 090817.909 INFO             PET00 UM time in u_model: 1981    9    1   23   40    0

Seems to be while writing the CICE restart file cice-mom.cice.r.0001-01-02-00000.nc. The MOM restart file cice-mom.mom6.r.0001-01-02-00000.nc is much larger but doesn't cause a delay.

The PET log files have huge amount of CICE related diagnostic output. E.g.

20230802 085927.464 INFO             PET00 ice_import tfrz_option = linear_salt, ktherm =        2
20230802 085927.472 ERROR            PET00 block:            1
20230802 085927.472 ERROR            PET00 lat:   -1.36401128638163
20230802 085927.472 ERROR            PET00 lon:  -0.688314068063793
20230802 085927.472 ERROR            PET00 rhoa:    1.29300000000000
....
20230802 085927.572 ERROR            PET00 lon:  -0.707948931563269
20230802 090811.307 INFO             PET00 ice_export called

Where does the time between the last of these messages and the call to ice_export go?

[MartinDix]

u-da593 is an n96 atmosphere with the OM3 build and ocean configuration. This doesn't show the slowdown.

20231019 064447.199 INFO             PET00 UM time in u_model: 1981    9    1   22    0    0
20231019 064454.327 INFO             PET00 UM time in u_model: 1981    9    1   22   20    0
20231019 064456.693 INFO             PET00 UM time in u_model: 1981    9    1   22   40    0
20231019 064510.369 INFO             PET00 UM time in u_model: 1981    9    1   23    0    0
20231019 064517.483 INFO             PET00 UM time in u_model: 1981    9    1   23   20    0
20231019 064519.862 INFO             PET00 UM time in u_model: 1981    9    1   23   40    0