add checks of io layout to cmeps driver

[anton-seaice]

This PR implements checks of the processors selected in the nuopc.runconfig file for the access-om3 payu driver. This ensures that the processors requested for normal model operations and parallel IO are within range of the CPUs set in config.yaml and each model "realm" uses processors for IO that are within the range of processors for that realm.

It adds tests for the min/max bounds of each parameter.

Contributes to COSIMA/access-om3#109

[pep8speaks]

Hello @anton-seaice! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file payu/models/cesm_cmeps.py:

Line 193:80: E501 line too long (86 > 79 characters)
Line 194:80: E501 line too long (90 > 79 characters)
Line 195:80: E501 line too long (86 > 79 characters)
Line 196:80: E501 line too long (90 > 79 characters)
Line 199:80: E501 line too long (83 > 79 characters)
Line 205:80: E501 line too long (90 > 79 characters)
Line 211:80: E501 line too long (83 > 79 characters)
Line 227:80: E501 line too long (89 > 79 characters)
Line 231:80: E501 line too long (81 > 79 characters)
Line 234:80: E501 line too long (87 > 79 characters)
Line 235:80: E501 line too long (86 > 79 characters)
Line 242:80: E501 line too long (92 > 79 characters)
Line 243:80: E501 line too long (88 > 79 characters)
Line 245:80: E501 line too long (97 > 79 characters)
Line 248:80: E501 line too long (93 > 79 characters)
Line 251:50: W291 trailing whitespace
Line 256:80: E501 line too long (90 > 79 characters)
Line 261:80: E501 line too long (93 > 79 characters)
Line 266:80: E501 line too long (84 > 79 characters)
Line 267:80: E501 line too long (89 > 79 characters)

• In the file test/models/access-om3/test_access_om3.py:

Line 9:80: E501 line too long (87 > 79 characters)
Line 101:80: E501 line too long (86 > 79 characters)
Line 115:80: E501 line too long (86 > 79 characters)
Line 141:80: E501 line too long (86 > 79 characters)
Line 153:80: E501 line too long (94 > 79 characters)
Line 180:80: E501 line too long (86 > 79 characters)
Line 189:80: E501 line too long (84 > 79 characters)
Line 218:80: E501 line too long (86 > 79 characters)
Line 226:80: E501 line too long (88 > 79 characters)
Line 280:80: E501 line too long (80 > 79 characters)

Comment last updated at 2024-09-18 06:20:40 UTC

[anton-seaice]

Apologies to the wrong minghangli !! @minghangli-uni, can you look at this please?

[coveralls]

coverage: 56.769% (+1.0%) from 55.76%
when pulling a90763b on om3_iolayout
into 2826621 on master.

[minghangli-uni]

Thanks @anton-seaice. I just did payu setup with a "wrong" pio setting, but it doesnt complain.

[minghangli-uni]

I've done a few tests on pio_root, pio_numiotasks and pio_stride. One ICE_modelio example below is based on a most recent merge of the 0.25 deg ryf configuration. The ncpus for cice is 96.

ICE_modelio::
     diro = ./log
     logfile = ice.log
     pio_async_interface = .false.
     pio_netcdf_format = nothing
     pio_numiotasks = 3
     pio_rearranger = 1
     pio_root = 100000
     pio_stride = 48
     pio_typename = netcdf4p
::

What I've found is,

1. pio_root does not appear to impact the I/O tasks as expected, which seems to be a bug. For example, setting pio_root to a very large value, such as 100000, does not affect the run.
2. Both pio_numiotasks and pio_stride function as intended, and their effects on the I/O tasks are properly reflected. The above example will throw me an error because the third IO task exceeds the range of available CICE cpus (i.e. 96)

[dougiesquire]

Thanks for adding this @anton-seaice. Now that we're starting to accumulate a few checks in the driver, I think it would be worth putting these in a new _qa_check (or something) method that's called from setup for clarity.

[minghangli-uni]

Thanks @anton-seaice for your work! I've left some comments. Can you please take a look at those?

[anton-seaice]

I've done a few tests on pio_root, pio_numiotasks and pio_stride. One ICE_modelio example below is based on a most recent merge of the 0.25 deg ryf configuration. The ncpus for cice is 96.

ICE_modelio::
     diro = ./log
     logfile = ice.log
     pio_async_interface = .false.
     pio_netcdf_format = nothing
     pio_numiotasks = 3
     pio_rearranger = 1
     pio_root = 100000
     pio_stride = 48
     pio_typename = netcdf4p
::

What I've found is,

1. `pio_root` does not appear to impact the I/O tasks as expected, which seems to be a bug. For example, setting `pio_root` to a very large value, such as 100000, does not affect the run.

pio_root in this example would be (almost silently) changed to 0 in the pio shared code. It's not a behaviour I love, I think it should just cause an error. Payu would reject this after this PR.

[anton-seaice]

@minghangli-uni - Are you able to review this please? I updated the initial comment with the scope

[minghangli-uni]

Thanks @anton-seaice for implementing this check. I have some comments and suggested changes that need to be addressed.

I havent checked the tests yet.

[anton-seaice]

@aidanheerdegen - this is ready to go. Can you or one of the release team want to review ?

[aidanheerdegen]

I have some questions and suggestions.

[aidanheerdegen]

Before I forget when this does get merged please either squash when merging, or rebase beforehand to clean up the commit history.

[aidanheerdegen]

LGTM. I have pinged @minghangli-uni because there are some unresolved conversations that he should check have been resolved and mark them as such. Then good to merge I think.

[minghangli-uni]

LGTM! Thanks @anton-seaice