Skip to content

Commit

Permalink
Move to contrib installation of spack-stack on Jet (#2878)
Browse files Browse the repository at this point in the history
Migrates Global Workflow to use contrib installation of spack-stack on
Jet.
Following the failure of the storage /lfs4 on Jet, the installation of
spack spack moved to /contrib.
All softwares relying on spack-stack on Jet needs update.

Resolves #2841 
Refs NOAA-EMC/gfs-utils#78
Refs NOAA-EMC/GSI#786
Refs NOAA-EMC/GSI-Monitor#143
Refs NOAA-EMC/GSI-utils#51
Refs ufs-community/UFS_UTILS#977
  • Loading branch information
InnocentSouopgui-NOAA authored Sep 30, 2024
1 parent e2f56f5 commit 8f0541c
Show file tree
Hide file tree
Showing 18 changed files with 34 additions and 31 deletions.
2 changes: 1 addition & 1 deletion docs/note_fixfield.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ They are saved locally on all platforms

Hera: /scratch1/NCEPDEV/global/glopara/fix
Orion/Hercules: /work/noaa/global/glopara/fix
Jet: /mnt/lfs4/HFIP/hfv3gfs/glopara/git/fv3gfs/fix
Jet: /mnt/lfs5/HFIP/hfv3gfs/glopara/FIX/fix
S4: /data/prod/glopara/fix

------------------------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion docs/source/components.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ Observation data, also known as dump data, is prepared in production and then ar

* Hera: ``/scratch1/NCEPDEV/global/glopara/dump``
* Orion/Hercules: ``/work/noaa/rstprod/dump``
* Jet: ``/mnt/lfs4/HFIP/hfv3gfs/glopara/dump``
* Jet: ``/mnt/lfs5/HFIP/hfv3gfs/glopara/dump``
* WCOSS2: ``/lfs/h2/emc/global/noscrub/emc.global/dump``
* S4: ``/data/prod/glopara/dump``

Expand Down
4 changes: 2 additions & 2 deletions docs/source/init.rst
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Warm-start cycled w/ coupled (S2S) model C48 atmosphere C48 enkf (80 members) 5
Hera: /scratch1/NCEPDEV/global/glopara/data/ICSDIR/C48C48mx500
Orion/Hercules: /work/noaa/global/glopara/data/ICSDIR/C48C48mx500
WCOSS2: /lfs/h2/emc/global/noscrub/emc.global/data/ICSDIR/C48C48mx500
Jet: /lfs4/HFIP/hfv3gfs/glopara/data/ICSDIR/C48C48mx500
Jet: /lfs5/HFIP/hfv3gfs/glopara/data/ICSDIR/C48C48mx500
AWS: https://noaa-nws-global-pds.s3.amazonaws.com/index.html#data/ICSDIR/C48C48mx500

Start date = 2021032312
Expand Down Expand Up @@ -227,7 +227,7 @@ Forecast-only P8 prototype initial conditions are made available to users on sup
WCOSS2: /lfs/h2/emc/global/noscrub/emc.global/IC/COUPLED
HERA: /scratch1/NCEPDEV/climate/role.ufscpara/IC
ORION/Hercules: /work/noaa/global/glopara/data/ICSDIR/prototype_ICs
JET: /mnt/lfs4/HFIP/hfv3gfs/glopara/data/ICSDIR/prototype_ICs
JET: /mnt/lfs5/HFIP/hfv3gfs/glopara/data/ICSDIR/prototype_ICs
S4: /data/prod/glopara/coupled_ICs

These locations are known within the workflow via paths set in ``parm/config/config.coupled_ic``.
Expand Down
2 changes: 1 addition & 1 deletion modulefiles/module_base.jet.lua
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ setenv("UTILROOT",(os.getenv("prod_util_ROOT") or "None"))
prepend_path("MODULEPATH", pathJoin("/lfs5/HFIP/hfv3gfs/glopara/git/prepobs/v" .. (os.getenv("prepobs_run_ver") or "None"), "modulefiles"))
load(pathJoin("prepobs", (os.getenv("prepobs_run_ver") or "None")))

prepend_path("MODULEPATH", pathJoin("/lfs4/HFIP/hfv3gfs/glopara/git/Fit2Obs/v" .. (os.getenv("fit2obs_ver") or "None"), "modulefiles"))
prepend_path("MODULEPATH", pathJoin("/lfs5/HFIP/hfv3gfs/glopara/git/Fit2Obs/v" .. (os.getenv("fit2obs_ver") or "None"), "modulefiles"))
load(pathJoin("fit2obs", (os.getenv("fit2obs_ver") or "None")))

whatis("Description: GFS run environment")
2 changes: 1 addition & 1 deletion modulefiles/module_gwsetup.jet.lua
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Load environment to run GFS workflow setup scripts on Jet

load(pathJoin("rocoto"))

prepend_path("MODULEPATH", "/mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/modulefiles/Core")
prepend_path("MODULEPATH", "/contrib/spack-stack/spack-stack-1.6.0/envs/gsi-addon-intel/install/modulefiles/Core")

local stack_intel_ver=os.getenv("stack_intel_ver") or "2021.5.0"
local python_ver=os.getenv("python_ver") or "3.11.6"
Expand Down
2 changes: 1 addition & 1 deletion parm/config/gfs/config.aero
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ case ${machine} in
AERO_INPUTS_DIR="/gpfs/f5/epic/proj-shared/global/glopara/data/gocart_emissions"
;;
"JET")
AERO_INPUTS_DIR="/lfs4/HFIP/hfv3gfs/glopara/data/gocart_emissions"
AERO_INPUTS_DIR="/lfs5/HFIP/hfv3gfs/glopara/data/gocart_emissions"
;;
*)
echo "FATAL ERROR: Machine ${machine} unsupported for aerosols"
Expand Down
2 changes: 1 addition & 1 deletion sorc/gfs_utils.fd
2 changes: 1 addition & 1 deletion sorc/gsi_monitor.fd
2 changes: 1 addition & 1 deletion sorc/link_workflow.sh
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ case "${machine}" in
"hera") FIX_DIR="/scratch1/NCEPDEV/global/glopara/fix" ;;
"orion") FIX_DIR="/work/noaa/global/glopara/fix" ;;
"hercules") FIX_DIR="/work/noaa/global/glopara/fix" ;;
"jet") FIX_DIR="/lfs4/HFIP/hfv3gfs/glopara/git/fv3gfs/fix" ;;
"jet") FIX_DIR="/lfs5/HFIP/hfv3gfs/glopara/FIX/fix" ;;
"s4") FIX_DIR="/data/prod/glopara/fix" ;;
"gaea") FIX_DIR="/gpfs/f5/ufs-ard/world-shared/global/glopara/data/fix" ;;
"noaacloud") FIX_DIR="/contrib/global-workflow-shared-data/fix" ;;
Expand Down
2 changes: 1 addition & 1 deletion sorc/ufs_model.fd
Submodule ufs_model.fd updated 73 files
+1 −1 CDEPS-interface/CDEPS
+0 −3 CDEPS-interface/cdeps_files.cmake
+1 −1 CMEPS-interface/CMEPS
+1 −1 FV3
+1 −1 MOM6-interface/MOM6
+5 −12 build.sh
+1 −1 tests/bl_date.conf
+8 −10 tests/ci/Jenkinsfile.combined
+2 −8 tests/compile.sh
+5 −4 tests/default_vars.sh
+3 −0 tests/error-test.conf
+2 −4 tests/fv3_conf/cpld_docn_dice.IN
+31 −31 tests/logs/OpnReqTests_control_p8_hera.log
+87 −79 tests/logs/OpnReqTests_cpld_control_nowave_noaero_p8_hera.log
+15 −15 tests/logs/OpnReqTests_regional_control_hera.log
+263 −340 tests/logs/RegressionTests_acorn.log
+278 −278 tests/logs/RegressionTests_derecho.log
+278 −278 tests/logs/RegressionTests_gaea.log
+367 −422 tests/logs/RegressionTests_hera.log
+371 −372 tests/logs/RegressionTests_hercules.log
+247 −247 tests/logs/RegressionTests_jet.log
+277 −277 tests/logs/RegressionTests_orion.log
+256 −254 tests/logs/RegressionTests_wcoss2.log
+1 −0 tests/parm/control.nml.IN
+1 −0 tests/parm/control_ca.nml.IN
+1 −0 tests/parm/control_csawmg.nml.IN
+1 −0 tests/parm/control_flake.nml.IN
+1 −1 tests/parm/control_gdas.nml.IN
+1 −0 tests/parm/control_gfsv17.nml.IN
+1 −0 tests/parm/control_noahmp.nml.IN
+1 −0 tests/parm/csawmg3shoc127.nml.IN
+1 −0 tests/parm/csawmgshoc.nml.IN
+1 −1 tests/parm/datm.streams.IN
+1 −0 tests/parm/global_control.nml.IN
+1 −0 tests/parm/input_global_hafs.nml.IN
+1 −0 tests/parm/input_global_nest.nml.IN
+1 −0 tests/parm/input_global_parent.nml.IN
+1 −0 tests/parm/input_nest02.nml.IN
+1 −0 tests/parm/input_nest_hafs.nml.IN
+1 −0 tests/parm/input_regional_hafs.nml.IN
+1 −0 tests/parm/rap.nml.IN
+1 −0 tests/parm/regional.nml.IN
+1 −0 tests/parm/regional_atmaq.nml.IN
+1 −0 tests/parm/regional_rrfs_a.nml.IN
+1 −0 tests/parm/regional_wofs.nml.IN
+1 −0 tests/parm/rrfs_conus13km_hrrr.nml.IN
+14 −4 tests/parm/ufs.configure.s2s_esmf.IN
+1 −0 tests/parm/wam_v17.nml.IN
+0 −5 tests/rt.conf
+1 −0 tests/rt.sh
+40 −161 tests/rt_utils.sh
+10 −1 tests/run_compile.sh
+128 −16 tests/run_test.sh
+1 −116 tests/test_changes.list
+4 −0 tests/tests/control_c48.v2.sfc_timeout
+2 −2 tests/tests/datm_cdeps_3072x1536_cfsr
+1 −1 tests/tests/datm_cdeps_bulk_cfsr
+3 −3 tests/tests/datm_cdeps_bulk_gefs
+1 −1 tests/tests/datm_cdeps_ciceC_cfsr
+1 −1 tests/tests/datm_cdeps_control_cfsr
+1 −1 tests/tests/datm_cdeps_control_cfsr_faster
+3 −3 tests/tests/datm_cdeps_control_gefs
+1 −1 tests/tests/datm_cdeps_debug_cfsr
+3 −3 tests/tests/datm_cdeps_gfs
+3 −3 tests/tests/datm_cdeps_iau_gefs
+1 −1 tests/tests/datm_cdeps_multiple_files_cfsr
+1 −1 tests/tests/datm_cdeps_mx025_cfsr
+3 −3 tests/tests/datm_cdeps_mx025_gefs
+1 −1 tests/tests/datm_cdeps_restart_cfsr
+3 −3 tests/tests/datm_cdeps_stochy_gefs
+0 −45 tests/tests/regional_ifi_2threads
+0 −45 tests/tests/regional_ifi_control
+0 −45 tests/tests/regional_ifi_decomp
2 changes: 1 addition & 1 deletion sorc/ufs_utils.fd
Submodule ufs_utils.fd updated 35 files
+2 −2 driver_scripts/driver_grid.jet.sh
+2 −2 fix/link_fixdirs.sh
+5 −14 modulefiles/build.hera.gnu.lua
+1 −1 modulefiles/build.jet.intel.lua
+13 −11 modulefiles/build.wcoss2.intel.lua
+2 −0 reg_tests/chgres_cube/13km.conus.nam.grib2.sh
+2 −0 reg_tests/chgres_cube/13km.conus.rap.grib2.sh
+2 −0 reg_tests/chgres_cube/13km.na.gfs.ncei.grib2.sh
+2 −0 reg_tests/chgres_cube/25km.conus.gfs.grib2.sh
+2 −0 reg_tests/chgres_cube/25km.conus.gfs.pbgrib2.sh
+2 −0 reg_tests/chgres_cube/3km.conus.hrrr.gfssdf.grib2.sh
+2 −0 reg_tests/chgres_cube/3km.conus.hrrr.newsfc.grib2.sh
+2 −2 reg_tests/chgres_cube/driver.jet.sh
+4 −3 reg_tests/chgres_cube/driver.wcoss2.sh
+15 −21 reg_tests/cpld_gridgen/RegressionTests_jet.intel.log
+1 −1 reg_tests/cpld_gridgen/parm/grid.nml.IN
+8 −4 reg_tests/cpld_gridgen/rt.sh
+2 −2 reg_tests/global_cycle/driver.jet.sh
+2 −2 reg_tests/global_cycle/driver.wcoss2.sh
+2 −2 reg_tests/grid_gen/driver.jet.sh
+1 −1 reg_tests/grid_gen/driver.orion.sh
+2 −2 reg_tests/grid_gen/driver.wcoss2.sh
+5 −5 reg_tests/ice_blend/driver.jet.sh
+11 −11 reg_tests/ocnice_prep/RegressionTests_jet.intel.log
+9 −5 reg_tests/ocnice_prep/rt.sh
+2 −2 reg_tests/snow2mdl/driver.jet.sh
+2 −2 reg_tests/weight_gen/driver.jet.sh
+2 −2 reg_tests/weight_gen/driver.wcoss2.sh
+1 −1 sorc/chgres_cube.fd/program_setup.F90
+2 −0 sorc/fre-nctools.fd/shared_lib/mosaic_util.h
+18 −2 sorc/global_cycle.fd/cycle.f90
+1 −1 sorc/machine-setup.sh
+1 −1 ush/global_cycle.sh
+2 −2 util/gdas_init/set_fixed_files.sh
+1 −1 util/weight_gen/run.jet.sh
2 changes: 1 addition & 1 deletion ush/detect_machine.sh
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ if [[ -d /lfs/h3 ]]; then
elif [[ -d /lfs/h1 && ! -d /lfs/h3 ]]; then
# We are on NOAA TDS Acorn
MACHINE_ID=acorn
elif [[ -d /mnt/lfs1 ]]; then
elif [[ -d /mnt/lfs5 ]]; then
# We are on NOAA Jet
MACHINE_ID=jet
elif [[ -d /scratch1 ]]; then
Expand Down
1 change: 0 additions & 1 deletion ush/forecast_predet.sh
Original file line number Diff line number Diff line change
Expand Up @@ -548,7 +548,6 @@ FV3_predet(){
${NCP} "${PARMgfs}/post/gefs/postxconfig-NT-gefs-f00.txt" "${DATA}/postxconfig-NT_FH00.txt"
fi
fi

}

# Disable variable not used warnings
Expand Down
4 changes: 2 additions & 2 deletions versions/build.jet.ver
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
export stack_intel_ver=2021.5.0
export stack_impi_ver=2021.5.1
export spack_env=gsi-addon-dev
export spack_env=gsi-addon-intel
source "${HOMEgfs:-}/versions/spack.ver"
export spack_mod_path="/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-${spack_stack_ver}/envs/${spack_env}/install/modulefiles/Core"
export spack_mod_path="/contrib/spack-stack/spack-stack-${spack_stack_ver}/envs/${spack_env}/install/modulefiles/Core"
8 changes: 6 additions & 2 deletions versions/run.jet.ver
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
export stack_intel_ver=2021.5.0
export stack_impi_ver=2021.5.1
export spack_env=gsi-addon-dev-rocky8
export spack_env=gsi-addon-intel

export hpss_ver=
export ncl_ver=6.6.2
Expand All @@ -11,4 +11,8 @@ export gempak_ver=7.4.2
export perl_ver=5.38.0

source "${HOMEgfs:-}/versions/spack.ver"
export spack_mod_path="/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-${spack_stack_ver}/envs/${spack_env}/install/modulefiles/Core"
export spack_mod_path="/contrib/spack-stack/spack-stack-${spack_stack_ver}/envs/${spack_env}/install/modulefiles/Core"

# Local version of TC_tracker
export ens_tracker_ver=v1.1.15.7

2 changes: 1 addition & 1 deletion workflow/hosts.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ def detect(cls):
machine = 'HERA'
elif os.path.exists('/work/noaa'):
machine = socket.gethostname().split("-", 1)[0].upper()
elif os.path.exists('/lfs4/HFIP'):
elif os.path.exists('/lfs5/HFIP'):
machine = 'JET'
elif os.path.exists('/lfs/f1'):
machine = 'WCOSS2'
Expand Down
22 changes: 11 additions & 11 deletions workflow/hosts/jet.yaml
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
BASE_GIT: '/lfs4/HFIP/hfv3gfs/glopara/git'
DMPDIR: '/lfs4/HFIP/hfv3gfs/glopara/dump'
BASE_GIT: '/lfs5/HFIP/hfv3gfs/glopara/git'
DMPDIR: '/lfs5/HFIP/hfv3gfs/glopara/dump'
BASE_DATA: '/lfs5/HFIP/hfv3gfs/glopara/data'
BASE_IC: '/mnt/lfs4/HFIP/hfv3gfs/glopara/data/ICSDIR'
PACKAGEROOT: '/lfs4/HFIP/hfv3gfs/glopara/nwpara'
COMINsyn: '/lfs4/HFIP/hfv3gfs/glopara/com/gfs/prod/syndat'
HOMEDIR: '/lfs4/HFIP/hfv3gfs/${USER}'
STMP: '/lfs4/HFIP/hfv3gfs/${USER}/stmp'
PTMP: '/lfs4/HFIP/hfv3gfs/${USER}/ptmp'
BASE_IC: '/mnt/lfs5/HFIP/hfv3gfs/glopara/data/ICSDIR'
PACKAGEROOT: '/lfs5/HFIP/hfv3gfs/glopara/nwpara'
COMINsyn: '/lfs5/HFIP/hfv3gfs/glopara/com/gfs/prod/syndat'
HOMEDIR: '/lfs5/HFIP/hfv3gfs/${USER}'
STMP: '/lfs5/HFIP/hfv3gfs/${USER}/stmp'
PTMP: '/lfs5/HFIP/hfv3gfs/${USER}/ptmp'
NOSCRUB: $HOMEDIR
ACCOUNT: hfv3gfs
SCHEDULER: slurm
Expand All @@ -25,6 +25,6 @@ ATARDIR: '/NCEPDEV/${HPSS_PROJECT}/1year/${USER}/${machine}/scratch/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
MAKE_ACFTBUFR: 'NO'
SUPPORTED_RESOLUTIONS: ['C384', 'C192', 'C96', 'C48']
COMINecmwf: /mnt/lfs4/HFIP/hfv3gfs/glopara/data/external_gempak/ecmwf
COMINnam: /mnt/lfs4/HFIP/hfv3gfs/glopara/data/external_gempak/nam
COMINukmet: /mnt/lfs4/HFIP/hfv3gfs/glopara/data/external_gempak/ukmet
COMINecmwf: /mnt/lfs5/HFIP/hfv3gfs/glopara/data/external_gempak/ecmwf
COMINnam: /mnt/lfs5/HFIP/hfv3gfs/glopara/data/external_gempak/nam
COMINukmet: /mnt/lfs5/HFIP/hfv3gfs/glopara/data/external_gempak/ukmet

0 comments on commit 8f0541c

Please sign in to comment.