ufs-community · zach1221 · Dec 14, 2023 · Aug 31, 2023 · Aug 31, 2023 · Sep 13, 2023
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -4,7 +4,7 @@ PLEASE DO NOT MODIFY THE TEMPLATE BEYOND FILLING OUT THE PROPER SECTIONS -->
 <!--  Please complete all items in list. -->
 - [ ] I have linked PR's from all sub-components involved in section below. <!-- PLEASE DO NOT LINK SUBCOMPONENT ISSUES -->
 - [ ] I am confirming reviews are completed in ALL sub-component PR's.
-- [ ] I have run the full RT suite on either Hera/Cheyenne AND have attached the log to this PR below this line:
+- [ ] I have run the full RT suite on either Hera/Hercules AND have attached the log to this PR below this line:
   - LOG: 
 - [ ] I have added the list of all failed regression tests to "Anticipated changes" section.
 - [ ] I have filled out all sections of the template.
@@ -79,7 +79,7 @@ PLEASE DO NOT MODIFY THE TEMPLATE BEYOND FILLING OUT THE PROPER SECTIONS -->
   - [ ] Hercules
   - [ ] Jet
   - [ ] Gaea
-  - [ ] Cheyenne
+  - [ ] Derecho
 - WCOSS2
   - [ ] Dogwood/Cactus
   - [ ] Acorn

diff --git a/doc/UsersGuide/source/AutomatedTesting.rst b/doc/UsersGuide/source/AutomatedTesting.rst
@@ -91,7 +91,7 @@ On supported HPC systems, a :term:`cron job` runs the ``start_rt_auto.sh`` bash
 This script checks the HPC name and sets certain python paths. Then, it runs ``rt_auto.py``, 
 which uses the Github API (through pyGitHub) to check the labels on pull requests to 
 ``ufs-weather-model``. If a PR label matches the HPC name 
-(e.g., hera-intel-RT or cheyenne-gnu-BL), the label provides the HPC  
+(e.g., hera-intel-RT or derecho), the label provides the HPC  
 with the compiler and job information to run a test or task on the machine. 
 If no PR label matches HPC name, the script exits.
 

diff --git a/doc/UsersGuide/source/BuildingAndRunning.rst b/doc/UsersGuide/source/BuildingAndRunning.rst
@@ -10,7 +10,7 @@ Supported Platforms & Compilers
 Before running the Weather Model (:term:`WM`), users should determine which of the 
 :ref:`levels of support <SupportedPlatforms>` 
 is applicable to their system. Generally, Level 1 & 2 systems are restricted to those with access 
-through NOAA and its affiliates. These systems are named (e.g., Hera, Orion, Cheyenne). 
+through NOAA and its affiliates. These systems are named (e.g., Hera, Orion, Derecho). 
 Level 3 & 4 systems include certain personal computers or non-NOAA-affiliated HPC systems. 
 The prerequisite software libraries for building the WM already exist in a centralized location on Level 1/preconfigured 
 systems, so users may skip directly to :ref:`getting the data <GetData>` and downloading the code. 
@@ -89,23 +89,23 @@ the data required to run the WM RTs are already available in the following locat
 .. _DataLocations:
 .. table:: Data Locations for Level 1 & 2 Systems
 
-   +--------------+-----------------------------------------------------+
-   | Machine      | File location                                       |
-   +==============+=====================================================+
-   | Cheyenne     | /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT  |
-   +--------------+-----------------------------------------------------+
-   | Gaea         | /lustre/f2/pdata/ncep_shared/emc.nemspara/RT        |
-   +--------------+-----------------------------------------------------+
-   | Hera         | /scratch1/NCEPDEV/nems/emc.nemspara/RT              |
-   +--------------+-----------------------------------------------------+
-   | Jet          | /mnt/lfs4/HFIP/hfv3gfs/role.epic/RT                 |
-   +--------------+-----------------------------------------------------+
-   | Orion        | /work/noaa/nems/emc.nemspara/RT                     |
-   +--------------+-----------------------------------------------------+
-   | S4           | /data/prod/emc.nemspara/RT                          |
-   +--------------+-----------------------------------------------------+ 
-   | WCOSS2       | /lfs/h2/emc/nems/noscrub/emc.nems/RT                |
-   +--------------+-----------------------------------------------------+ 
+   +--------------+--------------------------------------------------------+
+   | Machine      | File location                                          |
+   +==============+========================================================+
+   | Derecho      | /glade/derecho/scratch/epicufsrt/ufs-weather-model/RT  |
+   +--------------+--------------------------------------------------------+
+   | Gaea         | /lustre/f2/pdata/ncep_shared/emc.nemspara/RT           |
+   +--------------+--------------------------------------------------------+
+   | Hera         | /scratch1/NCEPDEV/nems/emc.nemspara/RT                 |
+   +--------------+--------------------------------------------------------+
+   | Jet          | /mnt/lfs4/HFIP/hfv3gfs/role.epic/RT                    |
+   +--------------+--------------------------------------------------------+
+   | Orion        | /work/noaa/nems/emc.nemspara/RT                        |
+   +--------------+--------------------------------------------------------+
+   | S4           | /data/prod/emc.nemspara/RT                             |
+   +--------------+--------------------------------------------------------+ 
+   | WCOSS2       | /lfs/h2/emc/nems/noscrub/emc.nems/RT                   |
+   +--------------+--------------------------------------------------------+ 
 
 For Level 3-4 systems, the data must be added to the user's system. 
 Publicly available RT data is available in the `UFS WM Data Bucket <https://registry.opendata.aws/noaa-ufs-regtests/>`__. 

diff --git a/doc/UsersGuide/source/FAQ.rst b/doc/UsersGuide/source/FAQ.rst
@@ -29,14 +29,14 @@ The steps are:
          cd tests
 
    #. Find a configure (``*.conf``) file that contains the machine and compiler you are using. For this
-      example, the Intel compiler on Cheyenne is used.  To create a custom configure file, two lines are
+      example, the Intel compiler on Derecho is used.  To create a custom configure file, two lines are
       needed:  a ``COMPILE`` line and a ``RUN`` line.   The ``COMPILE`` line should contain the name
-      of the machine and compiler ``cheyenne.intel`` and the desired ``SUITES`` for the build.  Choose a
+      of the machine and compiler ``derecho.intel`` and the desired ``SUITES`` for the build.  Choose a
       ``RUN`` line under this ``COMPILE`` command that uses the desired ``SUITE``.  For example:
 
       .. code-block:: console
 
-         COMPILE | 32BIT=Y CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta,FV3_GFS_v15p2_no_nsst,FV3_GFS_v16beta_no_nsst                     | standard    | cheyenne.intel | fv3
+         COMPILE | 32BIT=Y CCPP=Y STATIC=Y SUITES=FV3_GFS_v15p2,FV3_GFS_v16beta,FV3_GFS_v15p2_no_nsst,FV3_GFS_v16beta_no_nsst                     | standard    | derecho.intel | fv3
          RUN     | fv3_ccpp_gfs_v16beta                                                                                                           | standard    |                | fv3         |
 
       Put these two lines into a file called ``my_test.conf``.  The parameters used in this run can be
@@ -49,16 +49,16 @@ The steps are:
 
       .. code-block:: console
 
-         if [[ $MACHINE_ID = cheyenne.* ]]; then stanza:
+         if [[ $MACHINE_ID = derecho.* ]]; then stanza:
          ...
          dprefix=/glade/scratch
 
-      This works for Cheyenne, since ``$USER/FV3_RT`` will be appended.  Also check that ``RTPWD``
+      This works for Derecho, since ``$USER/FV3_RT`` will be appended.  Also check that ``RTPWD``
       points to a diretory that exists:
 
       .. code-block:: console
 
-         if [[ $MACHINE_ID = cheyenne.* ]]; then
+         if [[ $MACHINE_ID = derecho.* ]]; then
             RTPWD=${RTPWD:-$DISKNM/ufs-public-release-20200224/${COMPILER^^}}
 
    #. Run the ``rt.sh`` script from the ``tests`` directory:

diff --git a/doc/UsersGuide/source/Introduction.rst b/doc/UsersGuide/source/Introduction.rst
@@ -36,7 +36,7 @@ for those elements, are listed below:
 
 The UFS Weather Model is currently included in two UFS Application releases: The UFS Short-Range Weather (:term:`SRW`) Application v2.0.0 release (June 2022) and the UFS Medium Range Weather Application (:term:`MRW`) v1.1.0 release (October 2020). These UFS Apps also contain pre- and post-processing components, a comprehensive build system, and workflows for configuration and execution of the application. The SRW App v2.0.0 documentation and details can be found `here <https://ufs-srweather-app.readthedocs.io/en/release-public-v2/>`__. The MRW App v1.1.0 documentation and details can be found `here <https://ufs-mrweather-app.readthedocs.io/en/ufs-v1.1.0>`__.
 
-The UFS WM code is portable and can be used with Linux or Mac operating systems and with Intel or GNU compilers. It has been tested on a variety of platforms widely used by atmospheric scientists, such as the NOAA Research Hera system, the National Center for Atmospheric Research (:term:`NCAR`) Cheyenne system, the National Science Foundation Stampede system, and Mac laptops.
+The UFS WM code is portable and can be used with Linux or Mac operating systems and with Intel or GNU compilers. It has been tested on a variety of platforms widely used by atmospheric scientists, such as the NOAA Research Hera system, the National Center for Atmospheric Research (:term:`NCAR`) Derecho system, the National Science Foundation Stampede system, and Mac laptops.
 
 .. note::
 
@@ -66,4 +66,4 @@ This WM User's Guide is organized as follows:
 
 Finally, :numref:`Chapters %s <Acronyms>` and :numref:`%s <Glossary>` contain a list of acronyms and a glossary, respectively.
 
-.. bibliography:: references.bib
+.. bibliography:: references.bib
diff --git a/modulefiles/ufs_cheyenne.gnu.lua b/modulefiles/ufs_cheyenne.gnu.lua
diff --git a/modulefiles/ufs_cheyenne.intel.lua b/modulefiles/ufs_cheyenne.intel.lua
diff --git a/modulefiles/ufs_derecho.intel.lua b/modulefiles/ufs_derecho.intel.lua
@@ -0,0 +1,30 @@
+help([[
+loads UFS Model prerequisites for NOAA Parallelworks/Intel
+]])
+
+setenv("LMOD_TMOD_FIND_FIRST","yes")
+prepend_path("MODULEPATH", "/lustre/desc1/scratch/epicufsrt/contrib/modulefiles")
+load("ecflow/5.8.4")
+load("mysql/8.0.33")
+
+setenv("LMOD_TMOD_FIND_FIRST","yes")
+prepend_path("MODULEPATH", "/lustre/desc1/scratch/epicufsrt/contrib/modulefiles_extra")
+prepend_path("MODULEPATH", "/glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.5.0/envs/unified-env/install/modulefiles/Core")
+
+unload("ncarcompilers")
+stack_intel_ver=os.getenv("stack_intel_ver") or "2021.10.0"
+load(pathJoin("stack-intel", stack_intel_ver))
+
+stack_impi_ver=os.getenv("stack_impi_ver") or "8.1.25"
+load(pathJoin("stack-cray-mpich", stack_impi_ver))
+
+cmake_ver=os.getenv("cmake_ver") or "3.26.3"
+load(pathJoin("cmake", cmake_ver))
+
+stack_python_ver=os.getenv("stack_python_ver") or "3.10.8"
+load(pathJoin("stack-python", stack_python_ver))
+
+setenv("CMAKE_Platform", "derecho.intel")
+load("ufs-weather-model-env")
+
+whatis("Description: UFS build environment")
diff --git a/tests/auto-jenkins/jobs/bl.py b/tests/auto-jenkins/jobs/bl.py
@@ -43,10 +43,10 @@ def set_directories(job_obj):
         blstore = '/work/noaa/epic/hercules/UFS-WM_RT'
         rtbldir = '/work/noaa/stmp/role-epic/stmp/role-epic/FV3_RT/'\
                  f'REGRESSION_TEST'  
-    elif job_obj.machine == 'cheyenne':
-        workdir = '/glade/scratch/epicufsrt/autort/jenkins/autort/pr'
-        blstore = '/glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/NEMSfv3gfs'
-        rtbldir = '/glade/scratch/epicufsrt/FV3_RT/'\
+    elif job_obj.machine == 'derecho':
+        workdir = '/glade/derecho/scratch/epicufsrt/autort/jenkins/autort/pr'
+        blstore = '/glade/derecho/scratch/epicufsrt/ufs-weather-model/RT/NEMSfv3gfs'
+        rtbldir = '/glade/derecho/scratch/epicufsrt/FV3_RT/'\
                  f'REGRESSION_TEST'
     else:
         logger.critical(f'Machine {job_obj.machine} is not supported for this job')

diff --git a/tests/auto-jenkins/jobs/rt.py b/tests/auto-jenkins/jobs/rt.py
@@ -33,8 +33,8 @@ def set_directories(job_obj):
         workdir = '/work/noaa/epic-ps/role-epic-ps/autort/tests/auto/pr'
     elif job_obj.machine == 'hercules':
         workdir = '/work/noaa/epic/role-epic/autort/tests/auto/pr'
-    elif job_obj.machine == 'cheyenne':
-        workdir = '/glade/scratch/epicufsrt/autort/jenkins/autort/pr'
+    elif job_obj.machine == 'derecho':
+        workdir = '/glade/derecho/scratch/epicufsrt/autort/jenkins/autort/pr'
     else:
         print(f'Machine {job_obj.machine} is not supported for this job')
         raise KeyError

diff --git a/tests/auto-jenkins/rt_auto_jenkins.py b/tests/auto-jenkins/rt_auto_jenkins.py
@@ -83,8 +83,8 @@ def delete_pr_dirs(each_pr, machine):
         workdir = '/work/noaa/epic-ps/role-epic-ps/autort/pr'
     elif machine == 'hercules':
         workdir = '/work/noaa/epic/role-epic/autort/pr'
-    elif machine == 'cheyenne':
-        workdir = '/glade/scratch/epicufsrt/autort/jenkins/autort/pr'
+    elif machine == 'derecho':
+        workdir = '/glade/derecho/scratch/epicufsrt/autort/jenkins/autort/pr'
     else:
         logging.error(f'Machine {machine} is not supported for this job')
         raise KeyError
@@ -120,8 +120,8 @@ def delete_rt_dirs(in_dir, machine, workdir):
         rt_dir = '/work/noaa/stmp/bcurtis/stmp/bcurtis/FV3_RT'
     elif machine == 'hercules':
         rt_dir = '/work/noaa/stmp/bcurtis/stmp/bcurtis/FV3_RT'
-    elif machine == 'cheyenne':
-        rt_dir = '/glade/scratch/epicufsrt/FV3_RT'
+    elif machine == 'derecho':
+        rt_dir = '/glade/derecho/scratch/epicufsrt/FV3_RT'
     else:
         logging.error(f'Machine {machine} is not supported for this job')
         raise KeyError
@@ -301,11 +301,11 @@ def setup_env():
     elif bool(re.match(re.compile('Hercules-login.+'), hostname)):
         machine = 'hercules'
         os.environ['ACCNR'] = 'epic'
-    elif bool(re.match(re.compile('cheyenne.+'), hostname)):
-        machine = 'cheyenne'
+    elif bool(re.match(re.compile('derecho.+'), hostname)):
+        machine = 'derecho'
         os.environ['ACCNR'] = 'SCSG0002'
     elif bool(re.match(re.compile('chadmin.+'), hostname)):
-        machine = 'cheyenne'
+        machine = 'derecho'
         os.environ['ACCNR'] = 'SCSG0002'
     else:
         raise KeyError(f'Hostname: {hostname} does not match '\

diff --git a/tests/auto-jenkins/start_rt_auto_jenkins.sh b/tests/auto-jenkins/start_rt_auto_jenkins.sh
@@ -22,9 +22,10 @@ elif [[ $HOSTNAME == tfe* ]]; then
 elif [[ $HOSTNAME == gaea* ]]; then
   export PATH=/lustre/f2/pdata/esrl/gsd/contrib/miniconda3/4.8.3/envs/ufs-weather-model/bin:$PATH
   export PYTHONPATH=/lustre/f2/pdata/esrl/gsd/contrib/miniconda3/4.8.3/lib/python3.8/site-packages
-elif [[ $HOSTNAME == *chadmin* ]] || [[ $HOSTNAME == *cheyenne* ]]; then
-  export MACHINE_ID=cheyenne
+elif [[ $HOSTNAME == *chadmin* ]] || [[ $HOSTNAME == *derecho* ]]; then
+  export MACHINE_ID=derecho
   export PATH=/glade/p/ral/jntp/tools/miniconda3/4.8.3/envs/ufs-weather-model/bin:/glade/p/ral/jntp/tools/miniconda3/4.8.3/bin:$PATH
+  export PATH=/glade/work/epicufsrt/contrib/derecho/rocoto/bin:$PATH
   export PYTHONPATH=/glade/p/ral/jntp/tools/miniconda3/4.8.3/envs/ufs-weather-model/lib/python3.8/site-packages:/glade/p/ral/jntp/tools/miniconda3/4.8.3/lib/python3.8/site-packages
 else
   echo "No Python Path for this machine."

diff --git a/tests/auto/jobs/bl.py b/tests/auto/jobs/bl.py
@@ -38,7 +38,7 @@ def set_directories(job_obj):
         blstore = '/work/noaa/nems/emc.nemspara/RT/NEMSfv3gfs'
         rtbldir = '/work/noaa/stmp/bcurtis/stmp/bcurtis/FV3_RT/'\
                  f'REGRESSION_TEST_{job_obj.compiler.upper()}'
-    elif job_obj.machine == 'cheyenne':
+    elif job_obj.machine == 'derecho':
         workdir = '/glade/scratch/dtcufsrt/autort/tests/auto/pr'
         blstore = '/glade/p/ral/jntp/GMTB/ufs-weather-model/RT/NEMSfv3gfs'
         rtbldir = '/glade/scratch/dtcufsrt/FV3_RT/'\

diff --git a/tests/auto/jobs/rt.py b/tests/auto/jobs/rt.py
@@ -22,7 +22,7 @@ def set_directories(job_obj):
         workdir = '/lustre/f2/pdata/ncep/emc.nemspara/autort/pr'
     elif job_obj.machine == 'orion':
         workdir = '/work/noaa/nems/emc.nemspara/autort/pr'
-    elif job_obj.machine == 'cheyenne':
+    elif job_obj.machine == 'derecho':
         workdir = '/glade/scratch/dtcufsrt/autort/tests/auto/pr'
     else:
         print(f'Machine {job_obj.machine} is not supported for this job')

diff --git a/tests/auto/rt_auto.py b/tests/auto/rt_auto.py
@@ -223,7 +223,7 @@ def setup_env():
     elif bool(re.match(re.compile('Orion-login.+'), hostname)):
         machine = 'orion'
     elif bool(re.match(re.compile('chadmin.+'), hostname)):
-        machine = 'cheyenne'
+        machine = 'derecho'
         os.environ['ACCNR'] = 'P48503002'
     else:
         raise KeyError(f'Hostname: {hostname} does not match '\

diff --git a/tests/auto/start_rt_auto.sh b/tests/auto/start_rt_auto.sh
@@ -17,8 +17,9 @@ elif [[ $HOSTNAME == gaea* ]]; then
   export PATH=/lustre/f2/pdata/esrl/gsd/contrib/miniconda3/4.8.3/envs/ufs-weather-model/bin:$PATH
   export PYTHONPATH=/lustre/f2/pdata/esrl/gsd/contrib/miniconda3/4.8.3/lib/python3.8/site-packages
 elif [[ $HOSTNAME == *chadmin* ]]; then
-  export MACHINE_ID=cheyenne
+  export MACHINE_ID=derecho
   export PATH=/glade/p/ral/jntp/tools/miniconda3/4.8.3/envs/ufs-weather-model/bin:/glade/p/ral/jntp/tools/miniconda3/4.8.3/bin:$PATH
+  export PATH=/glade/work/epicufsrt/contrib/derecho/rocoto/bin:$PATH
   export PYTHONPATH=/glade/p/ral/jntp/tools/miniconda3/4.8.3/envs/ufs-weather-model/lib/python3.8/site-packages:/glade/p/ral/jntp/tools/miniconda3/4.8.3/lib/python3.8/site-packages
 else
   echo "No Python Path for this machine."