Skip to content

Development and Testing

Jump to bottom
Alper Altuntas edited this page Feb 11, 2022 · 7 revisions

Contents

1. Review of the repository organization

Along with the core MOM6 repository, we maintain an interface repository, called MOM_interface. The core MOM6 repository is a fork of GFDL's MOM6 repository and is shared with the broader MOM6 community, whereas the interface repository is specific to CESM only. This page describes the development and testing workflows for both repositories.

While more information can be found in Repository Structure section, we note here that this two-tier repository organization allows us to separate concerns and streamline the implementation of CESM-specific parameterization choices, test suites, tools, and mechanisms without interfering with the core MOM6 repository shared with the broader MOM6 community.

2. Testing overview

  • CESM Test suites: The following two test suites are routinely tested to evaluate MOM6 pull requests and create tags to be incorporated in CESM releases. In summary, the testing workflow consists of (1) checking out the latest development branches, (2) running one of the test suites below to generate baseline results, (3) applying changes to MOM6 and/or MOM_interface repositories, (4) running the test suite again to compare against the baseline, and, if all tests pass, (5) submitting pull request(s) to NCAR's MOM6 and/or MOM_interface repositories.

    • aux_mom: A comprehensive test suite for creating MOM6 tags or merging pull requests that bring in substantial changes. This test suite consists of about 10 tests and takes about an hour to complete on cheyenne.
    • pr_mom: A lightweight test suite for pull requests with small changes, e.g., pull requests with only a couple of changed files and a few hundred lines at most. The author of the pull request must ensure that the brought in changes are covered by this test suite. Otherwise, the more comprehensive aux_mom test must be run. This test suite consists of about 4 tests and takes around 40 minutes to complete on cheyenne.

    Note: Currently, aux_mom and pr_mom test suites can be run on the following machines: Available machines. Collaborators working on other machines must extend the above test suites to their machines.

    See the following documentation for more information on CESM's testing infrastructure: https://esmci.github.io/cime/versions/master/html/users_guide/testing.html

  • Automated CI tests: Both MOM6 and MOM_interface GitHub repositories are equipped with Travis CI tests that are run automatically when a pull request is submitted. These tests are auxiliary lightweight sanity checks that can be run on a very few numbers of cores. In the case of MOM6, the tests include idealized and simple standalone MOM6 runs. In the case of MOM_interface, the tests include consistency and linting tests for the parameterizations database located in param_templates and doctests for the MOM_RPS module located in cime_config. For a pull request to be evaluated, all Travis CI tests muss pass.

3. Development and testing workflow

The following are the five steps of the development workflow for NCAR's MOM6 and MOM_interface repositories. Note that the instructions in this section are to serve as a general guideline which is to be adapted according to your local CESM sandbox configuration, personal conventions, and development needs.

Step 1: Check out the latest MOM6 and MOM_interface branches

  • Note: In the below instructions, it is assumed that your local CESM is in the current working directory, and is named CESM. It is also assumed that the remote alias origin points to NCAR's forks of MOM6 and MOM_interface repositories, whose URL's are https://github.com/NCAR/MOM6.git, and https://github.com/ESCOMP/MOM_interface.git, respectively. Otherwise, make sure to adapt below commands accordingly. (You can check your tracked remote repository aliases by running git remote -v inside your local git repository.)

  • If you don't have a local copy of a recent CESM sandbox, download CESM+MOM6 as described in the following section: Downloading CESM with MOM6

  • Run the following commands to check out the latest master branch of MOM_interface. Notice that MOM_interface repository is located in CESM/components and is named mom as per CESM conventions.

    cd CESM/components/mom
git fetch origin
git checkout master
git pull origin master

  • Similarly, run the following commands to check out the latest dev/master branch of the core MOM6 repository. Notice that the core MOM6 repository is in CESM/components/mom/MOM6.

    cd MOM6
git fetch origin
git pull origin dev/ncar
git checkout dev/ncar

Step 2: Create a test baseline

What is a baseline? A CESM baseline is a collection of model input parameter files and output history files. CESM testing tool may be instructed to generate baseline while running a test suite. Baselines are generated and maintained in order to diagnose and track any changes that may be introduced in the model input and/or solutions as a result of code changes.

When to create a baseline? Developers need to create a new baseline only after an answer changing commit gets introduced in MOM6 or MOM_interface. An old baseline may be re-used if an answer changing commit was not introduced since the time it was generated.

After having checked out the latest MOM_interface and MOM6 branches, you can create a baseline for aux_mom and/or pr_mom test suites using the create_test script of CESM. (Refer to Testing Overview section to determine which test suite to use.)

  • To create an aux_mom baseline on cheyenne, for example, run the following command. Make sure to change [YOUR-TEST-ROOT] and [YOUR-BASELINE_ROOT] in the below command with a test root path and a baseline root path of your choice. You will need to specify a new test root and a baseline root each time you create a new baseline. Therefore, it is advisable to create a tests and a baselines directories, possibly in your scratch space, since these can take large disk space and may need to be recreated once or twice a month.
qcmd -- CESM/cime/scripts/create_test --xml-category aux_mom --xml-machine cheyenne --test-root [YOUR-TEST-ROOT] --generate [YOUR-BASELINE_ROOT]

  • Once the tool finishes creating, building, and submitting all the tests, you can check out the status of all the tests by running cs.status tool placed in your test root. Note that the copy of cs.status tool in your test root will have a suffix with date/time and an identifier to distinguish a specific test suite instance from others.

    Example:

    cd [YOUR-TEST-ROOT]
./cs.status.20200316_122414_ih3z73  # Note: your copy of cs.status will have a different suffix

    The tool prints out the status of several steps and phases of each test within a test suite. For the ERI.T62_t061.GMOM test within aux_mom test suite, for example, the printed out status message should look like this:

    ERI.T62_t061.GMOM.cheyenne_intel (Overall: PASS) details:
  PASS ERI.T62_t061.GMOM.cheyenne_intel CREATE_NEWCASE
  PASS ERI.T62_t061.GMOM.cheyenne_intel XML
  PASS ERI.T62_t061.GMOM.cheyenne_intel SETUP
  PASS ERI.T62_t061.GMOM.cheyenne_intel SHAREDLIB_BUILD time=3
  PASS ERI.T62_t061.GMOM.cheyenne_intel NLCOMP
  PASS ERI.T62_t061.GMOM.cheyenne_intel MODEL_BUILD time=370
  PASS ERI.T62_t061.GMOM.cheyenne_intel SUBMIT
  PASS ERI.T62_t061.GMOM.cheyenne_intel RUN time=950
  PASS ERI.T62_t061.GMOM.cheyenne_intel COMPARE_base_hybrid
  PASS ERI.T62_t061.GMOM.cheyenne_intel COMPARE_base_rest
  PASS ERI.T62_t061.GMOM.cheyenne_intel GENERATE /glade/scratch/altuntas/cesm.baselines/cheyenne/aux_mom/200316
  PASS ERI.T62_t061.GMOM.cheyenne_intel MEMLEAK
  PASS ERI.T62_t061.GMOM.cheyenne_intel SHORT_TERM_ARCHIVER

    Note Refer to the following documentation for more information on test name syntax: https://esmci.github.io/cime/versions/master/html/users_guide/testing.html

Step 3: Apply your changes

  • First, check out a new branch off of the main development branch of the repository you are planning to make changes (i.e., MOM6 and/or MOM_interface). Make sure to name your new branch concisely and clearly, e.g., fix_kpp_smoothing, add_nuopc_cap, etc.

    git checkout -b [YOUR-BRANCH-NAME]

  • Apply your source code modifications and test them appropriately. After confirming that your code enhancements and/or bug fixes compile and run expectedly, commit your changes.

    git commit -m "[SUMMARIZE-YOUR-CHANGES]"

Step 4: Run a comparison test against the baseline

  • Re-run aux_mom or pr_mom test-suite, this time to compare against the baseline, as follows. Make sure to provide the same [YOUR-TEST-ROOT] and [YOUR-BASELINE_ROOT] you provided when generating the baseline. 
    qcmd -- CESM/cime/scripts/create_test --xml-category aux_mom --xml-machine cheyenne --test-root [YOUR-TEST-ROOT] --compare [YOUR-BASELINE_ROOT]
  • Once the tool finishes creating, building and submitting all the tests within the test suite, you can again check out the status of all the tests by running the most recent cs.status tool placed in your test root. For the ERI.T62_t061.GMOM test, for example, the printed out status message should look like this: 
    ERI.T62_t061.GMOM.cheyenne_intel (Overall: PASS) details:
  PASS ERI.T62_t061.GMOM.cheyenne_intel CREATE_NEWCASE
  PASS ERI.T62_t061.GMOM.cheyenne_intel XML
  PASS ERI.T62_t061.GMOM.cheyenne_intel SETUP
  PASS ERI.T62_t061.GMOM.cheyenne_intel SHAREDLIB_BUILD time=3
  PASS ERI.T62_t061.GMOM.cheyenne_intel NLCOMP
  PASS ERI.T62_t061.GMOM.cheyenne_intel MODEL_BUILD time=370
  PASS ERI.T62_t061.GMOM.cheyenne_intel SUBMIT
  PASS ERI.T62_t061.GMOM.cheyenne_intel RUN time=950
  PASS ERI.T62_t061.GMOM.cheyenne_intel COMPARE_base_hybrid
  PASS ERI.T62_t061.GMOM.cheyenne_intel COMPARE_base_rest
  PASS ERI.T62_t061.GMOM.cheyenne_intel BASELINE /glade/scratch/altuntas/cesm.baselines/cheyenne/aux_mom/200316
  PASS ERI.T62_t061.GMOM.cheyenne_intel MEMLEAK
  PASS ERI.T62_t061.GMOM.cheyenne_intel SHORT_TERM_ARCHIVER
    Notice above that the GENERATE phase is replaced with the BASELINE phase whose status indicates whether the results of the new test run is bitwise identical to the results of the baseline run. If no answer changes are expected, this phase must pass.

Step 5: Submit a pull request

After having committed your changes and run the test suite, you can submit a pull request to NCAR's forks of MOM6 and/or MOM_interface repositories.

[TODO: describe how to submit a PR]

4. Tagging workflow

[TODO: further describe how to tag MOM6 and MOM_interface]

Appendix

A. Adding a new MOM input file:

A new MOM6 input file may be added within CESM as follows:

  1. If your file is of type NetCDF4, convert it to cdf5.

  2. Make sure to add a timestamp to the file name, e.g., ocean_hgrid_180829.nc.

  3. Copy the input file to an appropriate subdirectory within your local input data directory, i.e., DIN_LOC_ROOT. On glade, this is /glade/p/cesmdata/cseg/inputdata.

  4. Add the filename entry to param_templates/MOM_input.yaml file. For example, the following is the grid file entry in MOM_input.yaml:

    GRID_FILE:
        description: |
            "Name of the file from which to read horizontal grid data."
        datatype: string
        value:
            $OCN_GRID == "gx1v6": "ocean_hgrid.nc"
            $OCN_GRID == "tx0.66v1": "ocean_hgrid_180829.nc"
            $OCN_GRID == "tx0.25v1": "ocean_hgrid.nc"
  1. Add an entry to param_templates/input_data_list.yaml file. This is to make sure that CESM automatically downloads the file from the SVN inputdata repository if it is missing in the local inputdata directory of another machine. For example, the following is the grid file entry in input_data_list.yaml:
    ocean_hgrid:
        $OCN_GRID == "gx1v6":    "${INPUTDIR}/ocean_hgrid.nc"
        $OCN_GRID == "tx0.66v1": "${INPUTDIR}/ocean_hgrid_180829.nc"
        $OCN_GRID == "tx0.25v1": "${INPUTDIR}/ocean_hgrid.nc"

Note the entry identifier, which is ocean_hgrid in the above example, can be any arbitrary (but unique) name.

  1. Within the param_templates directory, run the ./yaml_to_json.py script so as to propagate your yaml file changes to the corresponding json files.

  2. Test your changes by running a test case and confirm the file gets read in by MOM6.

  3. If the test is successful, import your new input file to the CESM SVN inputdata repository. On glade, this can be done by running the ./rimport tool. To import a single file, you can run the following commands:

cd /glade/p/cesmdata/cseg/inputdata
./rimport -file [PATH-TO-NEW-FILE]
  1. Test your changes on a different machine, e.g., on hobart. This is to make sure that the file was imported to the SVN repositoty correctly, and that CIME can succesfully download it on a new machine.

User Manual: MOM6 in CESM

Clone this wiki locally