Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[develop] Update modulefiles to use hdf/1.14.0 and netcdf/4.9.2-based software stacks on Tier 1 systems #889

Merged
merged 25 commits into from
Sep 15, 2023
Merged

[develop] Update modulefiles to use hdf/1.14.0 and netcdf/4.9.2-based software stacks on Tier 1 systems #889

Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
87cf706
Update Orion data stage location to new role-epic space
Aug 16, 2023
57605c3
modulefiles update for netcdf/4.9.2-based hpc-stacks on Tier-1 platofms
Aug 16, 2023
88e2d57
Merge branch 'ufs-community:develop' into update_modulefiles
natalie-perlin Aug 24, 2023
e887591
Gaea modulefile update, use intel-classic-2023.1.0
Aug 24, 2023
d27734d
Merge recente updates from ufs-srweater-model develop branch
Aug 24, 2023
b4df2d4
Merge recent updates from ufs-srweater-model develop branch
Aug 24, 2023
cd76ced
Merge recent updates from ufs-srweater-model develop branch
Aug 24, 2023
fba689c
Merge branch 'update_modulefiles' of github.com:natalie-perlin/ufs-sr…
Aug 25, 2023
f327b3e
increased task walltime request and maxtries=2 for getting external d…
Aug 25, 2023
cfd2263
Update jet.yaml, use xjet only for forecast tasks
natalie-perlin Aug 31, 2023
3b17e52
Merge branch 'update_modulefiles' into update_modulefiles2
Sep 2, 2023
61ab620
srw_common.lua update to use for both hpc-stack and spack-stack
Sep 2, 2023
4331b71
Merge remote-tracking branch 'origin2/develop' into update_modulefiles2
Sep 2, 2023
a121489
update srw_common.lua, modulefile for Gaea C4: use intel/2023.1.0 com…
Sep 5, 2023
46683ee
updating srw_common.lua
Sep 5, 2023
8000d8d
Merge branch 'ufs-community:update_modulefiles' into update_modulefiles
natalie-perlin Sep 5, 2023
1111dd9
update wflow_gaea modulefile
Sep 5, 2023
1d66243
Merge branch 'ufs-community:update_modulefiles' into update_modulefiles
natalie-perlin Sep 5, 2023
ad77d8b
Merge branch 'develop' into update_modulefiles
natalie-perlin Sep 8, 2023
768ac5a
Update srw_common.lua
natalie-perlin Sep 12, 2023
7e409ea
Update miniconda_regional_workflow_cmaq.lua
natalie-perlin Sep 12, 2023
5c44b06
Update plot_allvars.local.lua
natalie-perlin Sep 12, 2023
0eef53d
Update wflow_gaea.lua
natalie-perlin Sep 12, 2023
d9e2a96
gaea modulefiles update, keep configuration for the GSI at the moment
Sep 13, 2023
a756559
merge with develop
Sep 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 1 addition & 7 deletions modulefiles/build_cheyenne_gnu.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,13 @@ setenv("MKLROOT", "/glade/u/apps/opt/intel/2022.1/mkl/latest")
load(pathJoin("ncarcompilers", os.getenv("ncarcompilers_ver") or "0.5.0"))
unload("netcdf")

prepend_path("MODULEPATH","/glade/work/epicufsrt/contrib/hpc-stack/gnu11.2.0/modulefiles/stack")
prepend_path("MODULEPATH","/glade/work/epicufsrt/contrib/hpc-stack/gnu11.2.0_ncdf492/modulefiles/stack")
load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("hpc-gnu", os.getenv("hpc_gnu_ver") or "11.2.0"))
load(pathJoin("hpc-mpt", os.getenv("hpc_mpt_ver") or "2.25"))

load("srw_common")

load(pathJoin("g2", os.getenv("g2_ver") or "3.4.5"))
load(pathJoin("esmf", os.getenv("esmf_ver") or "8.3.0b09"))
load(pathJoin("netcdf", os.getenv("netcdf_ver") or "4.7.4"))
load(pathJoin("libpng", os.getenv("libpng_ver") or "1.6.37"))
load(pathJoin("pio", os.getenv("pio_ver") or "2.5.7"))
load(pathJoin("fms", os.getenv("fms_ver") or "2022.04"))
load(pathJoin("openblas", os.getenv("openblas_ver") or "0.3.23"))

unsetenv("MKLROOT")
Expand Down
9 changes: 1 addition & 8 deletions modulefiles/build_cheyenne_intel.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,13 @@ load(pathJoin("python", os.getenv("python_ver") or "3.7.9"))
load(pathJoin("ncarcompilers", os.getenv("ncarcompilers_ver") or "0.5.0"))
unload("netcdf")

prepend_path("MODULEPATH","/glade/work/epicufsrt/contrib/hpc-stack/intel2022.1/modulefiles/stack")
prepend_path("MODULEPATH","/glade/work/epicufsrt/contrib/hpc-stack/intel2022.1_ncdf492/modulefiles/stack")
load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("hpc-intel", os.getenv("hpc_intel_ver") or "2022.1"))
load(pathJoin("hpc-mpt", os.getenv("hpc_mpt_ver") or "2.25"))

load("srw_common")

load(pathJoin("g2", os.getenv("g2_ver") or "3.4.5"))
load(pathJoin("esmf", os.getenv("esmf_ver") or "8.3.0b09"))
load(pathJoin("netcdf", os.getenv("netcdf_ver") or "4.7.4"))
load(pathJoin("libpng", os.getenv("libpng_ver") or "1.6.37"))
load(pathJoin("pio", os.getenv("pio_ver") or "2.5.7"))
load(pathJoin("fms", os.getenv("fms_ver") or "2022.04"))

setenv("CMAKE_C_COMPILER","mpicc")
setenv("CMAKE_CXX_COMPILER","mpicpc")
setenv("CMAKE_Fortran_COMPILER","mpif90")
Expand Down
15 changes: 6 additions & 9 deletions modulefiles/build_gaea_intel.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,22 +7,15 @@ whatis([===[Loads libraries needed for building the UFS SRW App on Gaea ]===])

load(pathJoin("cmake", os.getenv("cmake_ver") or "3.20.1"))

prepend_path("MODULEPATH","/lustre/f2/dev/role.epic/contrib/hpc-stack/intel-classic-2022.0.2/modulefiles/stack")
prepend_path("MODULEPATH","/lustre/f2/dev/role.epic/contrib/hpc-stack/intel-classic-2023.1.0/modulefiles/stack")
load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("hpc-intel-classic", os.getenv("hpc_intel_classic_ver") or "2022.0.2"))
load(pathJoin("hpc-intel-classic", os.getenv("hpc_intel_classic_ver") or "2023.1.0"))
load(pathJoin("hpc-cray-mpich", os.getenv("hpc_cray_mpich_ver") or "7.7.20"))

load("srw_common")
-- Need at runtime
load("alps")

local MKLROOT="/opt/intel/oneapi/mkl/2022.0.2/"
prepend_path("LD_LIBRARY_PATH",pathJoin(MKLROOT,"lib/intel64"))
pushenv("MKLROOT", MKLROOT)

pushenv("GSI_BINARY_SOURCE_DIR", "/lustre/f2/dev/role.epic/contrib/GSI_data/fix/20230601")
pushenv("CRAYPE_LINK_TYPE","dynamic")

natalie-perlin marked this conversation as resolved.
Show resolved Hide resolved
setenv("CC","cc")
setenv("FC","ftn")
setenv("CXX","CC")
Expand All @@ -31,3 +24,7 @@ setenv("CMAKE_CXX_COMPILER","CC")
setenv("CMAKE_Fortran_COMPILER","ftn")
setenv("CMAKE_Platform","gaea.intel")

setenv("CFLAGS","-diag-disable=10441")
setenv("FFLAGS","-diag-disable=10441 -fp-model source")


6 changes: 3 additions & 3 deletions modulefiles/build_hera_gnu.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,16 @@ load(pathJoin("cmake", os.getenv("cmake_ver") or "3.20.1"))
gnu_ver=os.getenv("gnu_ver") or "9.2.0"
load(pathJoin("gnu", gnu_ver))

prepend_path("MODULEPATH", "/scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-9.2/modulefiles/stack")
prepend_path("MODULEPATH", "/scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-9.2_ncdf492/modulefiles/stack")

load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("hpc-gnu", os.getenv("hpc-gnu_ver") or "9.2"))
load(pathJoin("hpc-mpich", os.getenv("hpc-mpich_ver") or "3.3.2"))

load("srw_common")

load(pathJoin("nccmp", os.getenv("nccmp_ver") or "1.8.9"))
load(pathJoin("nco", os.getenv("nco_ver") or "4.9.3"))
load(pathJoin("nccmp", os.getenv("nccmp_ver") or "1.9.1.0"))
load(pathJoin("nco", os.getenv("nco_ver") or "5.0.6"))
load(pathJoin("openblas", os.getenv("openblas_ver") or "0.3.23"))

unsetenv("MKLROOT")
Expand Down
4 changes: 2 additions & 2 deletions modulefiles/build_hera_intel.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ load(pathJoin("intel", intel_ver))
impi_ver=os.getenv("impi_ver") or "2022.1.2"
load(pathJoin("impi", impi_ver))

prepend_path("MODULEPATH","/scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/intel-2022.1.2/modulefiles/stack")
prepend_path("MODULEPATH","/scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/intel-2022.1.2_ncdf492/modulefiles/stack")

load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("hpc-intel", os.getenv("hpc_intel_ver") or "2022.1.2"))
Expand All @@ -25,7 +25,7 @@ load(pathJoin("hpc-impi", os.getenv("hpc_impi_ver") or "2022.1.2"))
load("srw_common")

load(pathJoin("nccmp", os.getenv("nccmp_ver") or "1.8.9.0"))
load(pathJoin("nco", os.getenv("nco_ver") or "4.9.3"))
load(pathJoin("nco", os.getenv("nco_ver") or "5.0.6"))

setenv("CMAKE_C_COMPILER","mpiicc")
setenv("CMAKE_CXX_COMPILER","mpiicpc")
Expand Down
2 changes: 1 addition & 1 deletion modulefiles/build_jet_intel.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ load("sutils")

load(pathJoin("cmake", os.getenv("cmake_ver") or "3.20.1"))

prepend_path("MODULEPATH","/mnt/lfs4/HFIP/hfv3gfs/role.epic/hpc-stack/libs/intel-2022.1.2/modulefiles/stack")
prepend_path("MODULEPATH","/mnt/lfs4/HFIP/hfv3gfs/role.epic/hpc-stack/libs/intel-2022.1.2_ncdf492/modulefiles/stack")
load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("hpc-intel", os.getenv("hpc_intel_ver") or "2022.1.2"))
load(pathJoin("hpc-impi", os.getenv("hpc_impi_ver") or "2022.1.2"))
Expand Down
4 changes: 2 additions & 2 deletions modulefiles/build_orion_intel.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ load("noaatools")
load(pathJoin("cmake", os.getenv("cmake_ver") or "3.22.1"))
load(pathJoin("python", os.getenv("python_ver") or "3.9.2"))

prepend_path("MODULEPATH","/work/noaa/epic/role-epic/contrib/orion/hpc-stack/intel-2022.1.2/modulefiles/stack")
prepend_path("MODULEPATH","/work/noaa/epic/role-epic/contrib/orion/hpc-stack/intel-2022.1.2_ncdf492/modulefiles/stack")
load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("hpc-intel", os.getenv("hpc_intel_ver") or "2022.1.2"))
load(pathJoin("hpc-impi", os.getenv("hpc_impi_ver") or "2022.1.2"))

load("srw_common")

load(pathJoin("nccmp", os.getenv("nccmp_ver") or "1.8.9.0"))
load(pathJoin("nco", os.getenv("nco_ver") or "4.9.3"))
load(pathJoin("nco", os.getenv("nco_ver") or "5.0.6"))

setenv("CMAKE_C_COMPILER","mpiicc")
setenv("CMAKE_CXX_COMPILER","mpiicpc")
Expand Down
22 changes: 11 additions & 11 deletions modulefiles/srw_common.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
load("jasper/2.0.25")
load("zlib/1.2.11")
load_any("png/1.6.35", "libpng/1.6.37")
load_any("jasper/2.0.25","jasper/2.0.32")
load_any("zlib/1.2.11","zlib/1.2.13")
load("libpng/1.6.37")

load_any("netcdf/4.7.4", "netcdf-c/4.7.4")
load_any("netcdf/4.7.4", "netcdf-fortran/4.5.4")
load_any("pio/2.5.7", "parallelio/2.5.2")
load_any("esmf/8.3.0b09", "esmf/8.2.0")
load("fms/2022.04")
load_any("netcdf/4.9.2", "netcdf-c/4.9.2")
load_any("netcdf/4.9.2", "netcdf-fortran/4.6.0")
load_any("pio/2.5.10","parallelio/2.5.10")
load("esmf/8.4.2")
load("fms/2023.01")

load("bufr/11.7.0")
load("bacio/2.4.1")
Expand All @@ -19,14 +19,14 @@ load("w3emc/2.9.2")

load_any("gftl-shared/v1.5.0", "gftl-shared/1.5.0")
load_any("yafyaml/v0.5.1", "yafyaml/0.5.1")
load_any("mapl/2.22.0-esmf-8.3.0b09", "mapl/2.11.0-esmf-8.2.0")
load("mapl/2.35.2-esmf-8.4.2")

load("nemsio/2.5.4")
load("sfcio/1.4.1")
load("sigio/2.3.2")
load("w3nco/2.4.1")
load("wrf_io/1.2.0")
load_any("wrf_io/1.2.0","wrf-io/1.2.0")

load("ncdiag/1.1.1")
--load("ncdiag/1.1.1")
load("ncio/1.1.2")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the Jenkins tests failed Friday night at the Build phase because GSI and rrfs_utils are still included in the Jenkins build. They will be removed at part of Christina's PR #893. Please reintroduce ncdiag until it is removed in PR #893.

load("wgrib2/2.0.8")
2 changes: 0 additions & 2 deletions modulefiles/wflow_gaea.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@ prepend_path("MODULEPATH","/lustre/f2/dev/role.epic/contrib/rocoto/modulefiles")
load("rocoto")
load("alps")

pushenv("MKLROOT", "/opt/intel/oneapi/mkl/2022.0.2/")
pushenv("GSI_BINARY_SOURCE_DIR", "/lustre/f2/dev/role.epic/contrib/GSI_data/fix/20230601")
setenv("PMI_NO_PREINITIALIZE","1")

if mode() == "load" then
Expand Down
10 changes: 8 additions & 2 deletions parm/wflow/coldstart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ default_task: &default_task
task_get_extrn_ics:
<<: *default_task
command: '&LOAD_MODULES_RUN_TASK_FP; "get_extrn_ics" "&JOBSdir;/JREGIONAL_GET_EXTRN_MDL_FILES"'
attrs:
cycledefs: forecast
maxtries: '2'
envars:
<<: *default_vars
ICS_OR_LBCS: ICS
Expand All @@ -32,7 +35,7 @@ task_get_extrn_ics:
partition: '{% if platform.get("PARTITION_HPSS") %}&PARTITION_HPSS;{% else %}None{% endif %}'
ppn: 1
queue: '&QUEUE_HPSS;'
walltime: 00:45:00
walltime: 01:30:00
dependency:
or:
and:
Expand All @@ -50,6 +53,9 @@ task_get_extrn_ics:
task_get_extrn_lbcs:
<<: *default_task
command: '&LOAD_MODULES_RUN_TASK_FP; "get_extrn_lbcs" "&JOBSdir;/JREGIONAL_GET_EXTRN_MDL_FILES"'
attrs:
cycledefs: forecast
maxtries: '2'
envars:
<<: *default_vars
ICS_OR_LBCS: LBCS
Expand All @@ -60,7 +66,7 @@ task_get_extrn_lbcs:
partition: '{% if platform.get("PARTITION_HPSS") %}&PARTITION_HPSS;{% else %}None{% endif %}'
ppn: 1
queue: '&QUEUE_HPSS;'
walltime: 00:45:00
walltime: 01:30:00
dependency:
or:
and:
Expand Down
2 changes: 1 addition & 1 deletion ush/machine/jet.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ platform:
DOMAIN_PREGEN_BASEDIR: /mnt/lfs4/HFIP/hfv3gfs/role.epic/UFS_SRW_data/develop/FV3LAM_pregen
PARTITION_DEFAULT: sjet,vjet,kjet,xjet
QUEUE_DEFAULT: batch
PARTITION_FCST: sjet,vjet,kjet,xjet
PARTITION_FCST: xjet
QUEUE_FCST: batch
PARTITION_HPSS: service
QUEUE_HPSS: batch
Expand Down
2 changes: 1 addition & 1 deletion ush/machine/orion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ platform:
TEST_EXTRN_MDL_SOURCE_BASEDIR: /work/noaa/epic/role-epic/contrib/UFS_SRW_data/develop/input_model_data
TEST_AQM_INPUT_BASEDIR: /work/noaa/epic/role-epic/contrib/UFS_SRW_data/develop/aqm_data
TEST_PREGEN_BASEDIR: /work/noaa/epic/role-epic/contrib/UFS_SRW_data/develop/FV3LAM_pregen
TEST_VX_FCST_INPUT_BASEDIR: '{{ "/work/noaa/epic-ps/role-epic-ps/UFS_SRW_data/develop/output_data/fcst_" }}{{ "ens" if (global.NUM_ENS_MEMBERS > 0) else "det" }}{{ "/{{workflow.PREDEF_GRID_NAME}}" }}{% raw %}{% endraw %}'
TEST_VX_FCST_INPUT_BASEDIR: '{{ "/work/noaa/epic/role-epic/contrib/UFS_SRW_data/develop/output_data/fcst_" }}{{ "ens" if (global.NUM_ENS_MEMBERS > 0) else "det" }}{{ "/{{workflow.PREDEF_GRID_NAME}}" }}{% raw %}{% endraw %}'
FIXaer: /work/noaa/epic/role-epic/contrib/UFS_SRW_data/develop/fix/fix_aer
FIXgsi: /work/noaa/epic/role-epic/contrib/UFS_SRW_data/develop/fix/fix_gsi
FIXgsm: /work/noaa/epic/role-epic/contrib/UFS_SRW_data/develop/fix/fix_am
Expand Down
Loading