Update for obsproc/v1.3 and new AFWA global snow filename (#2957)

Updates for the new `obsproc/v1.3` and AFWA global snow filename.
A new UFS_UTILS tag (`ops-gfsv16.3.20`) is used for the snow filename changes.

The prepobs package is also updated to `v1.2.0` but is internal to the upstream obsproc
package so only the `prepobs_run_ver` variables are updated for dev mode usage.

Additionally, workflow updates are made to allow the EMCSFC_SFC_PREP (sfcprep)
job to be run as part of the rocoto mesh in dev/emc mode. A new switch `DO_SFCPREP`
is added to `config.base.emc.dyn` and is set to "NO" by default. Setting it to "YES" 
will add the `sfcprep` job to the rocoto mesh ahead of the `prep` job and will generate
the snogrb files that are usually generated in ops and used via the GDA.

Refs #2913

Externals.cfg

@@ -29,7 +29,7 @@ protocol = git
 required = True
 
 [UFS_UTILS]
-tag = ops-gfsv16.3.0
+tag = ops-gfsv16.3.20
 local_path = sorc/ufs_utils.fd
 repo_url = https://github.com/ufs-community/UFS_UTILS.git
 protocol = git

docs/Release_Notes.md

@@ -1,10 +1,10 @@
-GFS V16.3.19 RELEASE NOTES
+GFS V16.3.20 RELEASE NOTES
 
 -------
 PRELUDE
 -------
 
-The WAFS is separated from the GFS and is now its own package in production as WAFS.v7.0.0.
+The upstream OBSPROC package is updated to v1.3. Along with this, the GFS is switching to use the AFWA global snow file due to the hemispheric snow files being phased out.
 
 IMPLEMENTATION INSTRUCTIONS
 ---------------------------
@@ -13,9 +13,9 @@ The NOAA VLab and the NOAA-EMC and NCAR organization spaces on GitHub are used t
 
 ```bash
 cd $PACKAGEROOT
-mkdir gfs.v16.3.19
-cd gfs.v16.3.19
-git clone -b EMC-v16.3.19 https://github.com/NOAA-EMC/global-workflow.git .
+mkdir gfs.v16.3.20
+cd gfs.v16.3.20
+git clone -b EMC-v16.3.20 https://github.com/NOAA-EMC/global-workflow.git .
 cd sorc
 ./checkout.sh -o
 ```
@@ -27,7 +27,7 @@ The checkout script extracts the following GFS components:
 | MODEL     | GFS.v16.3.1   | [email protected] |
 | GLDAS     | gldas_gfsv16_release.v.2.1.0 | [email protected] |
 | GSI       | gfsda.v16.3.12 | [email protected] |
-| UFS_UTILS | ops-gfsv16.3.0 | [email protected] |
+| UFS_UTILS | ops-gfsv16.3.20 | [email protected] |
 | POST      | upp_v8.3.0 | [email protected] |
 
 To build all the GFS components, execute:
@@ -49,141 +49,80 @@ cd ../ecf
 VERSION FILE CHANGES
 --------------------
 
-* `versions/run.ver` - change `version=v16.3.19` and `gfs_ver=v16.3.19`
+* `versions/run.ver` - change `version=v16.3.20`, `gfs_ver=v16.3.20`, and `obsproc_ver=v1.3`
 
 SORC CHANGES
 ------------
 
-The WAFS is no longer a submodule that is checked out within the GFS package.
-The `sorc/checkout.sh` and `Externals.cfg` checkout script no longer clone WAFS.
-The `sorc/build_all.sh` script no longer builds the WAFS code.
-The `sorc/build_gfs_wafs.sh` build script is deleted.
-The `sorc/link_fv3gfs.sh` script no longer links/copies WAFS files/execs.
+* New UFS_UTILS tag - `emcsfc_snow2mdl` program and associated scripts are updated to process global AFWA snow data
 
 JOBS CHANGES
 ------------
 
-All WAFS jobs are removed from the GFS ecFlow definition file, rocoto mesh, and `ush/ecflow/prod.yml`.
-Jobs removed:
-* `jgfs_atmos_wafs_gcip`
-* `jgfs_atmos_wafs_fFFF`
-* `jgfs_atmos_wafs_grib2`
-* `jgfs_atmos_wafs_grib2_0p25`
-* `jgfs_atmos_wafs_blending_0p25`
+* `jobs/JGLOBAL_ATMOS_EMCSFC_SFC_PREP` - new AFWA filename
 
 PARM/CONFIG CHANGES
 -------------------
 
-The following config files are deleted:
-* `parm/config/config.wafs`
-* `parm/config/config.wafsblending`
-* `parm/config/config.wafsblending0p25`
-* `parm/config/config.wafsgcip`
-* `parm/config/config.wafsgrib2`
-* `parm/config/config.wafsgrib20p25`
-
-* The `WAFSF` flag is removed from `parm/config/config.base.emc.dyn` and `parm/config/config.base.nco.static`.
-* All WAFS jobs are removed from platform env files, `parm/config/config.resources.emc.dyn`, and `parm/config/config.resources.nco.static`.
-* The WAFS jobs are removed from experiment setup.
-
-WAFS output is removed from the following transfer list files:
-* `parm/product/transfer_gfs_1.list`
-* `parm/product/transfer_gfs_7.list`
+* No changes from GFS v16.3.19
 
 SCRIPT CHANGES
 --------------
 
-The following WAFS rocoto scripts are removed:
-* `jobs/rocoto/wafs.sh`
-* `jobs/rocoto/wafsblending.sh`
-* `jobs/rocoto/wafsblending0p25.sh`
-* `jobs/rocoto/wafsgcip.sh`
-* `jobs/rocoto/wafsgrib2.sh`
-* `jobs/rocoto/wafsgrib20p25.sh`
-
-The following ecf scripts are removed from the GFS:
-* `ecf/scripts/gfs/atmos/post_processing/grib2_wafs/jgfs_atmos_wafs_blending.ecf`
-* `ecf/scripts/gfs/atmos/post_processing/grib2_wafs/jgfs_atmos_wafs_blending_0p25.ecf`
-* `ecf/scripts/gfs/atmos/post_processing/grib2_wafs/jgfs_atmos_wafs_grib2.ecf`
-* `ecf/scripts/gfs/atmos/post_processing/grib2_wafs/jgfs_atmos_wafs_grib2_0p25.ecf`
-* `ecf/scripts/gfs/atmos/post_processing/grib_wafs/jgfs_atmos_wafs_master.ecf`
-* `ecf/scripts/gfs/atmos/post_processing/jgfs_atmos_wafs_gcip.ecf`
-
-The WAFS is removed from `ecf/setup_ecf_links.sh`.
-
-The WAFS output is removed from archival (`ush/hpssarch_gen.sh`).
+* No changes from GFS v16.3.19
 
 FIX CHANGES
 -----------
 
-* `fix/product/wafs_admin_msg` - removed
+* No changes from GFS v16.3.19
 
 MODULE CHANGES
 --------------
 
-Modules needed by WAFS are removed from `modulefiles/module_base.wcoss_dell_p3`.
+* No changes from GFS v16.3.19
 
 CHANGES TO FILE AND FILE SIZES
 ------------------------------
 
-The following files will no longer be produced within the GFS COM:
-* `gfs.tCCz.awf_0p25.fFFF.grib2` - renamed to `wafs.tCCz.awf.0p25.fFFF.grib2` in WAFSv7
-* `gfs.tCCz.awf_grb45fFF.grib2` - renamed to `wafs.tCCz.awf_grid45.fFFF.grib2` in WAFSv7
-* `wmo/grib2.tCCz.awf_grbfFF.45` - renamed to `wmo/grib2.wafs.tCCz.awf_grid45.fFFF` in WAFSv7
-* `gfs.tCCz.control.wafsblending_0p25`
-* `gfs.tCCz.wafs.0p25.anl` - renamed to `wafs.tCCz.0p25.anl.grib2` in WAFSv7
-* `gfs.tCCz.wafs.0p25.anl.idx`
-* `gfs.tCCz.wafs_0p25.fFFF.grib2`
-* `gfs.tCCz.wafs_0p25.fFFF.grib2.idx`
-* `gfs.tCCz.wafs_0p25_unblended.fFFF.grib2` - renamed to `WAFS_0p25_unblended_YYYYMMDDHHfFFF.grib2` in WAFSv7
-* `gfs.tCCz.wafs_0p25_unblended.fFFF.grib2.idx`
-* `gfs.tCCz.wafs.grb2fFFF` - renamed to `wafs.tCCz.master.fFFF.grib2` in WAFSv7
-* `gfs.tCCz.wafs.grb2fFFF.idx`
-* `gfs.tCCz.wafs_grb45fFF.grib2` - renamed to `gfs.tCCz.wafs_grb45fFFF.grib2` in WAFSv7
-* `gfs.tCCz.wafs_grb45fFF.grib2.idx`
-* `wmo/grib2.tCCz.wafs_grbfFF.45` - renamed to `wmo/grib2.wafs.tCCz.awf_grid45.fFFF` in WAFSv7
-* `wmo/xtrn.wfsgfs0006.gfs_atmos_wafs_f30_00`
-* `gfs.tCCz.gcip.fFF.grib2` - renamed to `wafs.tCCz.gcip.fFFF.grib2` in WAFSv7
-* `WAFS_0p25_blended_YYYYMMDDHHf[06-48].grib2` - renamed to `WAFS_0p25_blended_ YYYYMMDDHHfFFF.grib2` in WAFSv7
-
-The following files will no longer be produced within the GFS COM
-and are being retired from the WAFS package:
-* `gfs.tCCz.wafs_icao.grb2fFFF`
-* `gfs.tCCz.wafs_icao.grb2fFFF.idx`
-* `wafs.tCCz.master.fFFF.grib2` where FFF is from 001 to 005
+No longer ingest:
+* `${RUN}.${cycle}.NPR.SNWN.SP.S1200.MESH16.grb` (`AFWA_NH_FILE`)
+* `${RUN}.${cycle}.NPR.SNWS.SP.S1200.MESH16.grb` (`AFWA_SH_FILE`)
+
+Now ingest:
+* `${RUN}.${cycle}.snow.usaf.grib2` (`AFWA_GLOBAL_FILE`)
 
 ENVIRONMENT AND RESOURCE CHANGES
 --------------------------------
 
-* No changes from GFS v16.3.18
+* No changes from GFS v16.3.19
 
 PRE-IMPLEMENTATION TESTING REQUIREMENTS
 ---------------------------------------
 
 * Which production jobs should be tested as part of this implementation?
-  * None
+  * emcsfc_sfc_prep job
 * Does this change require a 30-day evaluation?
   * No
 
 DISSEMINATION INFORMATION
 -------------------------
 
-* No changes from GFS v16.3.18
+* No changes from GFS v16.3.19
 
 HPSS ARCHIVE
 ------------
 
-* No changes from GFS v16.3.18
+* No changes from GFS v16.3.19
 
 JOB DEPENDENCIES AND FLOW DIAGRAM
 ---------------------------------
 
-* No changes from GFS v16.3.18
+* No changes from GFS v16.3.19
 
 DOCUMENTATION
 -------------
 
-* No changes from GFS v16.3.18
+* No changes from GFS v16.3.19
 
 PREPARED BY
 -----------

jobs/JGLOBAL_ATMOS_EMCSFC_SFC_PREP

@@ -8,7 +8,7 @@ date
 #############################
 # Source relevant config files
 #############################
-configs="base"
+configs="base sfcprep"
 export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
 config_path=${EXPDIR:-$NWROOT/gfs.${gfs_ver}/parm/config}
 for config in $configs; do
@@ -17,6 +17,13 @@ for config in $configs; do
     [[ $status -ne 0 ]] && exit $status
 done
 
+##########################################
+# Source machine runtime environment
+##########################################
+. $HOMEgfs/env/${machine}.env sfcprep
+status=$?
+[[ $status -ne 0 ]] && exit $status
+
 
 ##############################################
 # Obtain unique process id (pid) and make temp directory
@@ -64,8 +71,7 @@ export COMIN_m6hrs=${COMIN_m6hrs:-$(compath.py ${envir}/${NET}/${gfs_ver})/${RUN
 
 export IMS_FILE=${COMINobsproc}/${RUN}.${cycle}.imssnow96.grib2
 export FIVE_MIN_ICE_FILE=${COMINobsproc}/${RUN}.${cycle}.seaice.5min.grib2
-export AFWA_NH_FILE=${COMINobsproc}/${RUN}.${cycle}.NPR.SNWN.SP.S1200.MESH16.grb
-export AFWA_SH_FILE=${COMINobsproc}/${RUN}.${cycle}.NPR.SNWS.SP.S1200.MESH16.grb
+export AFWA_GLOBAL_FILE=${COMINobsproc}/${RUN}.${cycle}.snow.usaf.grib2
 
 export BLENDED_ICE_FILE=${BLENDED_ICE_FILE:-${RUN}.${cycle}.seaice.5min.blend.grb}
 export BLENDED_ICE_FILE_m6hrs=${BLENDED_ICE_FILE_m6hrs:-${COMIN_m6hrs}/${RUN}.${cycle_m6hrs}.seaice.5min.blend.grb}

jobs/rocoto/sfcprep.sh

@@ -0,0 +1,13 @@
+#!/bin/ksh -x
+
+###############################################################
+# Source FV3GFS workflow modules
+. $HOMEgfs/ush/load_fv3gfs_modules.sh
+status=$?
+[[ $status -ne 0 ]] && exit $status
+
+###############################################################
+# Execute the JJOB
+$HOMEgfs/jobs/JGLOBAL_ATMOS_EMCSFC_SFC_PREP
+status=$?
+exit $status

parm/config/config.base.emc.dyn

@@ -47,6 +47,9 @@ export NOSCRUB="@NOSCRUB@"
 # Base directories for various builds
 export BASE_GIT="@BASE_GIT@"
 
+# Toggle to turn on/off EMCSFC_SFC_PREP job
+export DO_SFCPREP="NO"    # SNOGRB dump file production
+
 # Toggle to turn on/off GFS downstream processing.
 export DO_BUFRSND="YES"    # BUFR sounding products
 export DO_GEMPAK="NO"      # GEMPAK products

parm/config/config.resources.emc.dyn

@@ -36,7 +36,16 @@ elif [[ "$machine" = "ORION" ]]; then
    export npe_node_max=40
 fi
 
-if [ $step = "prep" -o $step = "prepbufr" ]; then
+if [ $step = "sfcprep" ]; then
+
+    export wtime_sfcprep="00:08:00"
+    export npe_sfcprep=1
+    export nth_sfcprep=1
+    export npe_node_sfcprep=1
+    export NTASKS=$npe_sfcprep
+    export memory_sfcprep="2GB"
+
+elif [ $step = "prep" -o $step = "prepbufr" ]; then
 
     eval "export wtime_$step='00:45:00'"
     eval "export npe_$step=4"

parm/config/config.sfcprep

@@ -0,0 +1,20 @@
+#!/bin/ksh -x
+
+########## config.sfcprep ##########
+# Prep step specific
+
+echo "BEGIN: config.sfcprep"
+
+# Get task specific resources
+. $EXPDIR/config.resources sfcprep
+
+if [[ $RUN_ENVIR == "emc" ]]; then
+
+  export SENDCOM="YES"
+  export COMOUT=${COMOUTatmos}
+  export COMINobsproc=${DMPDIR}/${CDUMP}.${PDY}/${cyc}/atmos
+  export COMIN_m6hrs=${DMPDIR}/${GDUMP}.${gPDY}/${gcyc}/atmos
+
+fi
+
+echo "END: config.sfcprep"

sorc/checkout.sh

@@ -52,7 +52,7 @@ fi
 echo ufs_utils checkout ...
 if [[ ! -d ufs_utils.fd ]] ; then
     rm -f ${topdir}/checkout-ufs_utils.log
-    git clone --branch ops-gfsv16.3.0 https://github.com/ufs-community/UFS_UTILS ufs_utils.fd >> ${topdir}/checkout-ufs_utils.fd.log 2>&1
+    git clone --branch ops-gfsv16.3.20 https://github.com/ufs-community/UFS_UTILS ufs_utils.fd >> ${topdir}/checkout-ufs_utils.fd.log 2>&1
     cd ${topdir}
 else
     echo 'Skip.  Directory ufs_utils.fd already exists.'

ush/rocoto/setup_workflow.py

@@ -63,6 +63,7 @@ def main():
     steps = steps + wav_steps_gempak if _base.get('DO_GEMPAK', 'NO') == 'YES' else steps
     steps = steps + wav_steps_awips if _base.get('DO_AWIPS', 'NO') == 'YES' else steps
     steps = steps + ['wdqms'] if _base.get('DO_WDQMS', 'NO') == 'YES' else steps
+    steps = steps + ['sfcprep'] if _base.get('DO_SFCPREP', 'NO') == 'YES' else steps
 
     dict_configs = wfu.source_configs(configs, steps)
 
@@ -244,10 +245,13 @@ def get_gdasgfs_resources(dict_configs, cdump='gdas'):
     do_wdqms = base.get('DO_WDQMS', 'NO').upper()
     do_wave = base.get('DO_WAVE', 'NO').upper()
     do_wave_cdump = base.get('WAVE_CDUMP', 'BOTH').upper()
+    do_sfcprep = base.get('DO_SFCPREP', 'NO').upper()
     reservation = base.get('RESERVATION', 'NONE').upper()
 
-    #tasks = ['prep', 'anal', 'fcst', 'post', 'vrfy', 'arch']
-    tasks = ['prep', 'anal', 'analcalc']
+    if do_sfcprep in ['Y', 'YES']:
+        tasks = ['sfcprep', 'prep', 'anal', 'analcalc']
+    else:
+        tasks = ['prep', 'anal', 'analcalc']
 
     if cdump in ['gdas']:
         tasks += ['analdiag']
@@ -412,14 +416,29 @@ def get_gdasgfs_tasks(dict_configs, cdump='gdas'):
     do_wdqms = base.get('DO_WDQMS', 'NO').upper()
     do_wave = base.get('DO_WAVE', 'NO').upper()
     do_wave_cdump = base.get('WAVE_CDUMP', 'BOTH').upper()
+    do_sfcprep = base.get('DO_SFCPREP', 'NO').upper()
     dumpsuffix = base.get('DUMP_SUFFIX', '')
     gridsuffix = base.get('SUFFIX', '')
 
     dict_tasks = OrderedDict()
 
+    # sfcprep (emcsfc_sfc_prep)
+    if do_sfcprep in ['Y', 'YES']:
+        deps = []
+        dep_dict = {'type': 'metatask', 'name': f'{"gdas"}post', 'offset': '-06:00:00'}
+        deps.append(rocoto.add_dependency(dep_dict))
+        data = f'&DMPDIR;/{cdump}{dumpsuffix}.@Y@m@d/@H/atmos/{cdump}[email protected]'
+        dep_dict = {'type': 'data', 'data': data}
+        dependencies = rocoto.create_dependency(dep_condition='and', dep=deps)
+        task = wfu.create_wf_task('sfcprep', cdump=cdump, envar=envars, dependency=dependencies)
+        dict_tasks[f'{cdump}sfcprep'] = task
+
     # prep
     deps = []
-    dep_dict = {'type': 'metatask', 'name': f'{"gdas"}post', 'offset': '-06:00:00'}
+    if do_sfcprep in ['Y', 'YES']:
+        dep_dict = {'type': 'task', 'name': f'{cdump}sfcprep'}
+    else:
+        dep_dict = {'type': 'metatask', 'name': f'{"gdas"}post', 'offset': '-06:00:00'}
     deps.append(rocoto.add_dependency(dep_dict))
     data = f'&ROTDIR;/gdas.@Y@m@d/@H/atmos/[email protected]{gridsuffix}'
     dep_dict = {'type': 'data', 'data': data, 'offset': '-06:00:00'}

versions/hera.ver

@@ -2,8 +2,8 @@ export hpc_ver=1.2.0
 export hpc_intel_ver=18.0.5.274
 export hpc_impi_ver=2018.0.4
 
-export obsproc_run_ver=1.2.0
-export prepobs_run_ver=1.1.0
+export obsproc_run_ver=1.3.0
+export prepobs_run_ver=1.2.0
 
 export hpss_ver=hpss
 export prod_util_ver=1.2.2

versions/orion.ver

@@ -2,8 +2,8 @@ export hpc_ver=1.2.0
 export hpc_intel_ver=2018.4
 export hpc_impi_ver=2018.4
 
-export obsproc_run_ver=1.2.0
-export prepobs_run_ver=1.1.0
+export obsproc_run_ver=1.3.0
+export prepobs_run_ver=1.2.0
 
 export prod_util_ver=1.2.2
 export cmake_ver=3.22.1