Add handling of spatial simulations

[ccarouge]

Tests for the spatial/global configuration for CABLE offline should be the next configuration we add to the test suite (at least before CASA).

Sub-tasks for this issue include:

• Defining some data structure or class describe a spatial test. This will probably contain information such as the branch, the science configuration used for testing, and the met forcing.
• Organise model outputs produced from the spatial tests into a suitable directory structure.
• Running tests with payu. We will want to run with payu because of the following reasons:
  1. Encourage uptake of payu amongst users of CABLE.
  2. We will want to test CABLE online in benchcab and payu is routinely used for running an online configuration of CABLE.
  3. Using payu allows us to test longer running simulations easily (payu makes it easy to run a model multiple times and have state persist in the model via restart files).
• Merge in Add CABLE model driver payu-org/payu#314
• Add ability to build CABLE with MPI at runtime.
• Add payu in the benchcab conda environment #235

Spatial met forcing files and an example run directory are available here: /g/data/tm70/ccc561/CABLE/CABLE-as-ACCESS_Yingping. This can be used as inspiration for our run directories for the spatial tests. Note, if we use the same configurations we should:

• remove CASA from the run.
• make the namelist file as similar to the site runs. This isn't completely necessary but prevents us from eventual complications due to CABLE options.

[ccarouge]

For the spatial simulations, could we design it so that we have a base work directory (like bench_example) for each met. forcing source we want to use. This is the equivalent of the flux site met. forcing. And then, a list of namelist options is used to run various science configurations with each met. forcing.

Design questions to settle:

• how to run for several years for one met. forcing? Traditionally, met. forcing files are annual files which sets the length of the CABLE simulation. But we will want benchcab to run for several years for each task.
• how to deal with spin-up for the slow parts such as carbon stocks? Is it ok to provide a common starting point for all tasks?

[SeanBryan51]

For the directory structure, this is what I have as a plan so far:

runs/
├── spatial
│   └── tasks
│	├── <spatial-task-name> (a payu control / experiment directory)
│	└── ...
├── payu-laboratory
│   └── ...
└── fluxsite
    └── ...

The runs/spatial/tasks directory contains a bunch of payu control directories (or experiments) which are configured for each spatial task. Similar to the fluxsite case, each spatial task will have a unique realisation, science configuration and met forcing.

Note we have a separate payu-laboratory directory (this is so we keep all CABLE outputs produced by benchcab under the bench_example work directory). A custom laboratory path can be specified using the laboratory key in the payu config.yaml.

Also, the custom laboratory directory is in runs/payu-laboratory (i.e. outside the spatial directory). This is because we may want to have another configuration that runs with payu, e.g. online cable.

[SeanBryan51]

• how to run for several years for one met. forcing? Traditionally, met. forcing files are annual files which sets the length of the CABLE simulation. But we will want benchcab to run for several years for each task.

Running with payu allows us to use a different met forcing file for consecutive years/restarts (see https://github.com/CABLE-LSM/cable_example). However, we only have met forcing files up from 1901 to 1909 so the run will fail if we go beyond 1909. Should we add the ability to repeat the forcing every N years/restarts?

• how to deal with spin-up for the slow parts such as carbon stocks? Is it ok to provide a common starting point for all tasks?

I don't think I can answer this but it sounds reasonable to initialise CABLE with the same restart file. Would it beneficial to the user if they could use their own restart file?

It would also be great to have a test in benchcab that can verify the spinup is working correctly.

[SeanBryan51]

Some other design related questions:

1. Should we have some non-payu spatial tests?
   • There isn't much support for debugging models using the payu framework, removing a layer of abstraction will make things easier for the user to see how CABLE is being run.
   • We can still verify that CABLE is running correctly by running for a short time (e.g. 1 year). This would be enough to do bitwise comparisons between output files for example.
2. There is the issue of building with MPI using a custom build script. Should we add another build_script option specifically for building CABLE with MPI?
3. More of a comment, but since we are going with 1 PBS job per spatial task, we will probably have to use PBS job dependencies to run the bitwise comparison tasks on spatial outputs.

[ccarouge]

Note we have a separate payu-laboratory directory (this is so we keep all CABLE outputs produced by benchcab under the bench_example work directory). A custom laboratory path can be specified using the laboratory key in the payu config.yaml.

Also, the custom laboratory directory is

So why do we need to be able to set up a custom path for the laboratory? Do you have a use case in mind or it is just to give us the option if the use case appears afterwards?

In the last paragraph, I'm guessing you meant "standard laboratory directory" and not "custom". I am just mentioning it to make sure I understand correctly what you mean.

[SeanBryan51]

The main idea is to keep the "benchcab-payu" laboratory separate from the "standalone payu" laboratory. By default the laboratory is /scratch/${PROJECT}/${USER}/${MODEL}. A use case could be if we just want to reset the bench_example directory, we can rm -rf the runs directory without having to worry about any side effects that might have occurred outside of the bench_example directory.

[ccarouge]

2. There is the issue of building with MPI using a custom build script. Should we add another build_script option specifically for building CABLE with MPI?

Would the option be to get the command to run to build with MPI support?
An issue here is whether all historical scripts use the same way to choose an MPI compilation. Are all scripts using a command-line option or would some require hard-wired changes within the script? I don't know the answer. Should we only develop for the scripts we know of, which are using a command-line option (if I recall correctly)? In which case, we simply need an additional option to have the command that needs to be run.

[ccarouge]

The main idea is to keep the "benchcab-payu" laboratory separate from the "standalone payu" laboratory. By default the laboratory is /scratch/${PROJECT}/${USER}/${MODEL}. A use case could be if we just want to reset the bench_example directory, we can rm -rf the runs directory without having to worry about any side effects that might have occurred outside of the bench_example directory.

Sorry, it's my bad. I read "benchcab config.yaml" instead of "payu config.yaml", so I was wondering why we would want to let users specify their own path. I totally get the idea to set everything within the benchcab work directory. And that's much better that way.

[SeanBryan51]

Would the option be to get the command to run to build with MPI support?

Yep I see the issue here. We could do something similar to what is done in Github Actions to specify the commands? For example,

build_commands: |
    cd offline
    ./bespoke_build_script.sh --mpi

Also, should we run both fluxsite tests and spatial tests from the benchcab run command? In that case, we will probably need two custom build commands to be specified: one for serial build and one for MPI build. Or alternatively, we could just let the user specify the MPI build command and serial build command together like this:

build_commands: |
    cd offline
    ./bespoke_build_script.sh # build the serial executable
    ./bespoke_build_script.sh --clean # clean the build (if needed)
    ./bespoke_build_script.sh --mpi # build the MPI executable

Actually on second thought, this approach doesn't lend itself nicely for removing environment modules lines from the build script. It is probably sufficient to specify the arguments on top of the path to the build script:

build_script:
    - path: offline/bespoke_build_script.sh
    - args: [--mpi]

On third thought... 😆 if we prefer the Github Actions syntax, we could disable the module command by creating a mock module script which does nothing and prepend the mock script path to os.environ['PATH']. I remember exploring this idea back when I was looking at mocking system commands. This has the nice advantage of not having to create a temporary build script. See here for using python subprocess with a modified environment.

[ccarouge]

On third thought... 😆 if we prefer the Github Actions syntax, we could disable the module command by creating a mock module script which does nothing and prepend the mock script path to os.environ['PATH']. I remember exploring this idea back when I was looking at mocking system commands. This has the nice advantage of not having to create a temporary build script.

But this doesn't allow us to remove the "module load/add" lines from the build script or even worst "module purge", right? Hence opening the door to module inconsistencies.

[SeanBryan51]

But this doesn't allow us to remove the "module load/add" lines from the build script or even worst "module purge", right?

This shouldn't be a problem since we can let the mock take in any arbitrary arguments

[ccarouge]

Ah, I finally understood what you are proposing. You want to disable the module command for the command run in subprocess, not within benchcab. And it's not "a mock module script which does nothing and prepend ", but the mock script does nothing and we prepend its path to PATH when calling subprocess so the command in subprocess can find it. Now it makes sense to me.

About running fluxsites and spatial tests from one command, yes. But could be done in a future version. As you note, this would most likely require to clean the compilation between the 2. It would have to be done once we are sure the fluxsites tasks are done with.

Now that I understand your "third thought", I think the best is to ask for the exact commands and use the mock module script.

One question: is it easier if we have one key per command? Then it makes it clear people always need all 4 commands? Or is it enough to specify in the documentation? I am wondering if some people will think that the commands would be executed one after the other and worry if the "clean" option removes the built executable. Also, how do we ensure people give the commands in the correct order? What if someone gives MPI command first, single CPU command and then clean command? I can't see how we can check within benchcab which is which? I guess we can't check if people give the right command for a key either...

[SeanBryan51]

One question: is it easier if we have one key per command?

I don't think it's necessary as long as in the end we have a cable and cable-mpi executable in the offline directory, that way benchcab will find it. Since we are giving the user full control of what commands are being run, it's flexible in how the user can build the executables. For example, in the case that the clean command removes the executable, the user could have:

build_commands: |
    cd offline
    ./bespoke_build_script.sh # build the serial executable
    rm -rf .tmp # clean without removing the executable
    ./bespoke_build_script.sh --mpi # build the MPI executable

We would just have to make it clear to the user that when using custom build commands, benchcab expects all the executables it needs can be found in the offline directory for it to run the tests.

[ccarouge]

I am revisiting this considering the issue #130 around the parallelisation of the compilation. If we want to run all compilations at the same time, then should we have 2 checkout/clone of each source code's branch: one to compile serially and one to compile in parallel. And then you just get 2 commands: serial and MPI. In this case, should we have a key for each command? Or specify the order in the documentation?

[SeanBryan51]

I've added a comment on that issue here #130 (comment). For building legacy CABLE versions in parallel, we could defer this responsibility to the user? In theory, the user could background each process to run in parallel like this

build_commands: |
    cd offline
    ./serial_build &
    ./mpi_build &
    wait

[bschroeter]

Hey team! Please add your planning poker estimate with Zenhub @ccarouge @SeanBryan51