Skip to content

Commit

Permalink
Update project_charter.md with new environment names
Browse files Browse the repository at this point in the history
  • Loading branch information
@climbfuji
climbfuji committed Mar 22, 2024
1 parent e84c1b0 commit 43097d5
Show file tree
Hide file tree
Showing 3 changed files with 38 additions and 32 deletions.
6 changes: 5 additions & 1 deletion doc/source/KnownIssues.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ General

1. ``gcc@13`` (``gcc``, ``g++``, ``gfortran``) and ``apple-clang@15`` (``clang``, ``clang++``) not yet supported

Our software stack doesn't build with ``gcc@13`` yet. This is also true when combining the LLVM or Apple ``clang`` compiler with ``gfortran@13``. We also don't support the latest release of ``apple-clang@15`` yet.
Our software stack doesn't build with ``gcc@13`` yet. This is also true when combining the LLVM or Apple ``clang`` compiler with ``gfortran@13``. We also don't support the latest release of ``apple-clang@15`` with Xcode 15.3 yet, and with Xcode 15.0 a workaround is described in MISSING

2. Build errors for ``mapl@2.35.2`` with ``mpich@4.1.1``

Expand Down Expand Up @@ -53,6 +53,10 @@ NASA Discover

This can happen if a spack install is started in a ``screen`` session, because Discover puts the temporary data in directories like ``/gpfsm/dnb33/tdirs/login/discover13.29716.dheinzel``, which get wiped out after some time. Without ``screen``, this problem doesn't occur.

3. Insufficient diskspace when building ``py-pytorch``

This is because ``py-pytorch`` uses directory ``~/.ccache`` during the build, and the user's home directories have small quotas set. This problem can be avoided by creating a symbolic link from the home directory to a different place with sufficient quota: ``rm -fr ~/.ccache && ln -sf /path/to/dot_ccache_pytorch/ ~/.ccache``. It's probably a good idea to revert this hack after a successful installation.

==============================
NOAA Parallel Works
==============================
Expand Down
52 changes: 27 additions & 25 deletions doc/source/PreConfiguredSites.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,11 @@ On selected systems, developmental versions / release candidates are installed t
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| MSU | Hercules GCC+OpenMPI recommended | GCC | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/ue-gcc12-openmpi416`` | EPIC / JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Hercules | (GCC), Intel | ``/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env`` | EPIC / JCSDA |
| | Hercules | Intel | ``/work2/noaa/jcsda/dheinzel/spack-stack-1.7.0-rc1/envs/unified-env-*`` | EPIC / JCSDA |
| MSU +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Orion | GCC, Intel | ``/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/unified-env`` | EPIC / JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Discover SCU16 | GCC, Intel | ``/gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-20240228/envs/unified-env-*`` | JCSDA |
| | Discover SCU16 | GCC, Intel | ``/gpfsm/dnb55/projects/p01/s2127/dheinzel/sp170rc1/envs/unified-env-*`` | JCSDA |
| NASA +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Discover SCU17 | GCC, Intel | ``/gpfsm/dswdev/jcsda/spack-stack/scu17/spack-stack-20240228/envs/unified-env-*`` | JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
Expand All @@ -42,7 +42,7 @@ On selected systems, developmental versions / release candidates are installed t
| +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Jet | GCC, Intel | ``/mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env`` | EPIC / NOAA-EMC |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| | Narwhal | GCC, Intel | ``/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.6.0/envs/unified-env-*`` | JCSDA / NRL |
| | Narwhal | GCC, Intel | ``/p/work1/heinzell/spack-stack-1.7.0-rc1/envs/unified-env-*-craywrappers`` | JCSDA / NRL |
| +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| U.S. Navy (HPCMP) | Nautilus | Intel | ``/p/app/projects/NEPTUNE/spack-stack/spack-stack-1.6.0/envs/unified-env`` | JCSDA / NRL |
| +----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
Expand All @@ -60,6 +60,8 @@ On selected systems, developmental versions / release candidates are installed t
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| NOAA (RDHPCS) | RDHPCS Cloud (Parallel Works) | Intel | ``/contrib/spack-stack/spack-stack-1.6.0/envs/unified-env`` | EPIC / JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+
| NOAA (RDHPCS) | RDHPCS Gcloud only for testing | Intel | ``/contrib/spack-stack/spack-stack-1.7.0-rc1/envs/ue-intel-2021.3.0`` | JCSDA |
+---------------------+----------------------------------+-----------------+---------------------------------------------------------------------------------------------------------+-------------------------------+

For more information about a specific platform, please see the individual sections below.

Expand Down Expand Up @@ -125,31 +127,21 @@ The following is required for building new spack environments and for using spac
module load ecflow/5.8.4
module load git-lfs/3.1.2
For ``spack-stack-1.6.0`` with Intel, proceed with loading the following modules:
For ``spack-stack-1.7.0-rc1`` with Intel, proceed with loading the following modules:

.. code-block:: console
module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core
module use /work2/noaa/jcsda/dheinzel/spack-stack-1.7.0-rc1/envs/unified-env-intel-2021.9.0/install/modulefiles/Core
module load stack-intel/2021.9.0
module load stack-intel-oneapi-mpi/2021.9.0
module load stack-python/3.10.13
module available
For ``spack-stack-1.6.0`` with GNU, proceed with loading the following modules. Note that this environment is not recommended for GNU, an alternative installation using GNU+OpenMPI is available (see below).

.. code-block:: console
module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core
module load stack-gcc/12.2.0
module load stack-mvapich2/2.3.7
module load stack-python/3.10.13
module available
For ``spack-stack-1.6.0`` with GNU+OpenMPI, an alternative and recommended version is available. Load the following modules:
For ``spack-stack-1.7.0-rc1`` with GNU, proceed with loading the following modules:

.. code-block:: console
module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/ue-gcc12-openmpi416/install/modulefiles/Core
module use /work2/noaa/jcsda/dheinzel/spack-stack-1.7.0-rc1/envs/unified-env-gcc-12.2.0/install/modulefiles/Core
module load stack-gcc/12.2.0
module load stack-openmpi/4.1.6
module load stack-python/3.10.13
Expand Down Expand Up @@ -177,21 +169,21 @@ The following is required for building new spack environments and for using spac
module load miniconda/3.9.7
module load ecflow/5.8.4
For ``spack-stack-20240228`` with Intel, proceed with loading the following modules:
For ``spack-stack-1.7.0-rc1`` with Intel, proceed with loading the following modules:

.. code-block:: console
module use /gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-20240228/envs/unified-env-intel-2021.5.0/install/modulefiles/Core
module use /gpfsm/dnb55/projects/p01/s2127/dheinzel/sp170rc1/envs/unified-env-intel-2021.5.0/install/modulefiles/Core
module load stack-intel/2021.5.0
module load stack-intel-oneapi-mpi/2021.5.0
module load stack-python/3.10.13
module available
For ``spack-stack-20240228`` with GNU, proceed with loading the following modules:
For ``spack-stack-1.7.0-rc1`` with GNU, proceed with loading the following modules:

.. code-block:: console
module use /gpfsm/dswdev/jcsda/spack-stack/scu16/spack-stack-20240228/envs/unified-env-gcc-12.1.0/install/modulefiles/Core
module use /gpfsm/dnb55/projects/p01/s2127/dheinzel/sp170rc1/envs/unified-env-gcc-12.1.0/install/modulefiles/Core
module load stack-gcc/12.1.0
module load stack-openmpi/4.1.3
module load stack-python/3.10.13
Expand Down Expand Up @@ -255,15 +247,15 @@ With Intel, the following is required for building new spack environments and fo
module use /p/app/projects/NEPTUNE/spack-stack/modulefiles
module load ecflow/5.8.4
For ``spack-stack-1.6.0`` with Intel, proceed with loading the following modules:
For ``spack-stack-1.7.0-rc1`` with Intel, proceed with loading the following modules:

.. code-block:: console
# These extra steps are required for performance reason, ofi is about 30% slower than ucx
# Note we can't load craype-network-ucx for building spack-stack environments, must do here
module unload craype-network-ofi
module load craype-network-ucx
module use /p/app/projects/NEPTUNE/spack-stack/spack-stack-1.6.0/envs/unified-env-intel-2021.4.0/install/modulefiles/Core
module use /p/work1/heinzell/spack-stack-1.7.0-rc1/envs/unified-env-intel-2021.4.0-craywrappers/install/modulefiles/Core
module load stack-intel/2021.4.0
module load stack-cray-mpich/8.1.14
module load stack-python/3.10.13
Expand All @@ -287,15 +279,15 @@ With GNU, the following is required for building new spack environments and for
module use /p/app/projects/NEPTUNE/spack-stack/modulefiles
module load ecflow/5.8.4
For ``spack-stack-1.6.0`` with GNU, proceed with loading the following modules:
For ``spack-stack-1.7.0-rc1`` with GNU, proceed with loading the following modules:

.. code-block:: console
# These extra steps are required for performance reason, ofi is about 30% slower than ucx
# Note we can't load craype-network-ucx for building spack-stack environments, must do here
module unload craype-network-ofi
module load craype-network-ucx
module use /p/app/projects/NEPTUNE/spack-stack/spack-stack-1.6.0/envs/unified-env-gcc-10.3.0/install/modulefiles/Core
module use /p/work1/heinzell/spack-stack-1.7.0-rc1/envs/unified-env-gcc-10.3.0-craywrappers/install/modulefiles/Core
module load stack-gcc/10.3.0
module load stack-cray-mpich/8.1.14
module load stack-python/3.10.13
Expand Down Expand Up @@ -463,6 +455,16 @@ The following is required for building new spack environments and for using spac
module load ecflow/5.8.4
module load git-lfs/2.4.1
For testing, JCSDA installed ``spack-stack-1.7.0-rc1`` with Intel on Gcloud, which can be loaded as follows:

.. code-block:: console
module use /contrib/spack-stack/spack-stack-1.7.0-rc1/envs/ue-intel-2021.3.0/install/modulefiles/Core
module load stack-intel/2021.3.0
module load stack-intel-oneapi-mpi/2021.3.0
module load stack-python/3.10.13
module available
For ``spack-stack-1.6.0`` with Intel, proceed with loading the following modules:

.. code-block:: console
Expand Down
12 changes: 6 additions & 6 deletions project_charter.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -116,24 +116,24 @@ which case the work will be shared by the code managers.

#### Directory structure of spack-stack installs

/path/to/spack-stack/spack-stack-x.y.z/envs/unified-env-compiler-name-compiler-version/install/compiler-name/compiler-version/package-name-version-hash
/path/to/spack-stack/spack-stack-x.y.z/envs/ue-compiler-name-compiler-version/install/compiler-name/compiler-version/package-name-version-hash

_Example_
/Users/heinzell/prod/spack-stack-1.4.0/envs/unified-env-apple-clang-13.1.6/install/apple-clang/13.1.6/netcdf-c-4.9.2-vrrvi2u
/Users/heinzell/prod/spack-stack-1.4.0/envs/ue-apple-clang-13.1.6/install/apple-clang/13.1.6/netcdf-c-4.9.2-vrrvi2u

#### Auto-generated modules structure (no MPI dependency)

/path/to/spack-stack/spack-stack-x.y.z/envs/unified-env/install/modulefiles/compiler-name/compiler-version/package-name/package-version[.lua]
/path/to/spack-stack/spack-stack-x.y.z/envs/ue-compiler-name-compiler-version/install/modulefiles/compiler-name/compiler-version/package-name/package-version[.lua]

_Example_
/Users/heinzell/prod/spack-stack-1.4.0/envs/unified-env/install/modulefiles/apple-clang/13.1.6/sfcio/1.4.1.lua
/Users/heinzell/prod/spack-stack-1.4.0/envs/ue-apple-clang-13.1.6/install/modulefiles/apple-clang/13.1.6/sfcio/1.4.1.lua

#### Auto-generated modules structure (MPI dependency)

/path/to/spack-stack/spack-stack-x.y.z/envs/unified-env/install/modulefiles/mpi-name/mpi-version/compiler-name/compiler-version/package-name/package-version[.lua]
/path/to/spack-stack/spack-stack-x.y.z/envs/ue-compiler-name-compiler-version/install/modulefiles/mpi-name/mpi-version/compiler-name/compiler-version/package-name/package-version[.lua]

_Example_
/Users/heinzell/prod/spack-stack-1.4.0/envs/unified-env/install/modulefiles/openmpi/4.1.5/apple-clang/13.1.6/hdf5/1.14.0.lua
/Users/heinzell/prod/spack-stack-1.4.0/envs/ue-apple-clang-13.1.6/install/modulefiles/openmpi/4.1.5/apple-clang/13.1.6/hdf5/1.14.0.lua

#### Important points to remember

Expand Down

0 comments on commit 43097d5

Please sign in to comment.