Enable modules for packages not compiled with preferred compiler (update spack-stack setup-meta-modules); remove need for external `ecflow` #1257

climbfuji · 2024-08-20T22:15:20Z

Summary

Note. I pinky swear that I am going to rewrite that old setup-meta-modules extension that was clobbered together in an afternoon three years ago with the thinking "let's try something and then do it properly next month".

This PR makes the necessary changes to the site configs and the setup-meta-modules extension to build certain packages with other compilers than the preferred compiler and still be able to load the module for that package. We can only do this now that we have the concept of a preferred compiler in our environments. One assumption made here is that the MPI provider, if any, is compiled with the preferred compiler (I think this is a reasonable assumption to make).

Our use cases are:

bison needs to be compiled with gcc when the preferred compiler is oneapi. Strictly speaking, we don't need the bison module in this case, but I tested it on my laptop and it works.
ecflow and boost must be compiled with gcc when the preferred compiler is intel. This allows us to move away from external ecflow packages that don't work with the proposed update of Python to 3.11.7 (because the external ecflow was compiled with an old Python 3.9). In this case, we need the ecflow modulefile. I tested this on the Ubuntu CI runner and on Narwhal.

Caveat: I have not tested if this new capability works with packages that depend on MPI (which is compiled with the preferred compiler) but that get compiled with a different compiler (e.g. something like intel-oneapi-mpi/2021.12.0/gcc/11.2.0 when the packages using the preferred compiler would have intel-oneapi-mpi/2021.12.0/intel/2021.12.0).

Still todo:

Configure gcc compiler used as backend for Intel for Atlantis, Gaea C5, Gaea C6
- Atlantis deferred to a follow-up PR that also configures the oneAPI compiler
- Gaea C5 and Gaea C6 - @AlexanderRichert-NOAA @RatkoVasic-NOAA any last-minute changes for the gcc backend for Gaea C5 and C6 for this PR, or do you want to do this as a follow-up PR/when we roll out the release?
Remove external ecflow from packages.yaml and make sure every site has an external qt@5 in its packages.yaml
~~Fix unit test, or disable because we are going to rewrite the setup-meta-modules extension after the 1.8.0 release. Yes, this time for sure!~~ done

Testing

Tested for oneapi / bison on @climbfuji's laptop (unified environment)
Tested for intel / ecflow on Ubuntu CI runner and on Narwhal
More testing?

Applications affected

None (no changes to how applications are run)

Systems affected

All using Intel or oneAPI compilers

Dependencies

none

Issue(s) addressed

Link the issues addressed or resolved by this PR (use Fixes #??? for fully resolved issues)

Checklist

This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
These changes have been tested on some of the affected systems and applications.
~~All dependency PRs/issues have been resolved and this PR can be merged.~~

…cereal in ecflow

… ecflow, add external qt@5

…le for intel still works

…eature/boost_gcc4intel

… for principal compiler Intel

…n env using one principal (preferred) compiler

…6_64-intel.yaml

…eature/boost_gcc4intel

…ment

…eature/boost_gcc4intel

…mpilers.yaml if not already there and list in packages_intel.yaml (or packages.yaml if only an Intel build exists). Still todo: atlantis, gaea-c5, gaea-c6 (added notes)

…k/meta_modules.py

… .github/workflows/ubuntu-ci-x86_64-oneapi.yaml

…dules.py

climbfuji · 2024-08-23T15:00:32Z

spack-ext/lib/jcsda-emc/spack-stack/stack/meta_modules.py

+            logging.info("  ... ... appending {} to MODULEPATHS_SAVE".format(modulepath_save))
+            MODULEPATHS_SAVE.append(modulepath_save)
+
+            # For tcl modules remove the compiler prefices from the module contents


This block (remove the compiler prefices for tcl) got moved up, since it is needed for all compilers - not just the preferred compiler. This allows us to skip the remainder of the loop for compilers that are not the preferred compiler.

…tack/stack/meta_modules.py

RatkoVasic-NOAA

Successfully installed intel on Hercules.
Approved.

…eature/boost_gcc4intel

ashley314 · 2024-08-27T15:09:40Z

I was able to install on S4, although ecflow might not have successfully compiled with gcc? I also cannot figure out how to load the spack build ecflow instead of the version at /data/prod/jedi/spack-stack/.

climbfuji · 2024-08-27T15:56:06Z

@ashley314 Your issue may be that you still have an external ecflow in the S4 site config. You will want to remove that and make sure you have an external qt@5 instead. And you will want to remove the ecflow module from the list of excluded modules in the site config. I think @srherbener can help you (I'll be talking about the PR in the spack-stack meeting today).

AlexanderRichert-NOAA

This seems to work fine based on testing on my local machine (namely, loading stack-intel reveals gcc-built modules, e.g., boost).

climbfuji · 2024-08-27T20:44:38Z

This seems to work fine based on testing on my local machine (namely, loading stack-intel reveals gcc-built modules, e.g., boost).

Thanks for testing @AlexanderRichert-NOAA ! I'll wait for approval from JCSDA before I merge this.

ashley314 · 2024-08-27T21:14:07Z

@climbfuji thanks for the suggestions. I worked with @srherbener, we removed ecflow as an external package and qt was already declared. The exclude for ecflow also needed to be removed from the modules file. We were able to then install ecflow and made it through ldmod refresh with everything looking good. Ran the meta module script, but then when trying to load the jedi environment ecflow is still not showing up. Do you have any ideas on what else is missing?

Although still seeing only boost:

climbfuji · 2024-08-27T21:15:57Z

@climbfuji thanks for the suggestions. I worked with @srherbener, we removed ecflow as an external package and qt was already declared. The exclude for ecflow also needed to be removed from the modules file. We were able to then install ecflow and made it through ldmod refresh with everything looking good. Ran the meta module script, but then when trying to load the jedi environment ecflow is still not showing up. Do you have any ideas on what else is missing?
Although still seeing only boost:

I'll ping you in slack

…s correctly

…rst for UCX bug fixes

…i/spack-stack into feature/boost_gcc4intel

srherbener · 2024-08-28T14:33:27Z

configs/sites/tier1/gaea-c5/packages.yaml

@@ -1,6 +1,6 @@
 packages:
  all:
-    compiler:: [[email protected]]
+    compiler:: [[email protected]] # todo: add gcc here


Just want to double check. Is the intention to wait until after the 1.8.0 release to make these changes for Gaea?

We'll do this as part of the site config updates on the release branch and then bring it back to develop if that makes sense?

Sure, that makes sense. Happy to approve!

srherbener · 2024-08-28T15:14:45Z

configs/sites/tier1/gaea-c5/packages.yaml

@@ -1,6 +1,6 @@
 packages:
  all:
-    compiler:: [[email protected]]
+    compiler:: [[email protected]] # todo: add gcc here


Sure, that makes sense. Happy to approve!

climbfuji added 7 commits August 20, 2024 15:20

For Intel environments, build boost with %gcc to avoid problems with …

1331219

…cereal in ecflow

Temporarily move workflows out of the way to test ubuntu intel only

032671b

Update .github/workflows/ubuntu-ci-x86_64-intel.yaml: remove external…

eb8b401

… ecflow, add external qt@5

Add boost to module exclude list for tcl and lmod so that ecflow modu…

d127488

…le for intel still works

Merge branch 'develop' of https://github.com/jcsda/spack-stack into f…

284ccd8

…eature/boost_gcc4intel

Merge branch 'develop' of https://github.com/jcsda/spack-stack into f…

8705ecf

…eature/boost_gcc4intel

Merge branch 'develop' of https://github.com/jcsda/spack-stack into f…

68df671

…eature/boost_gcc4intel

ashley314 mentioned this pull request Aug 22, 2024

testing require gcc for boost #1267

Closed

3 tasks

climbfuji added 2 commits August 22, 2024 11:47

Revert boost module exclusion and require ecflow to be built with GCC…

40aa2e0

… for principal compiler Intel

[skip ci]

f34e6a9

climbfuji force-pushed the feature/boost_gcc4intel branch from e0c6d92 to 5b75c23 Compare August 22, 2024 18:29

[skip ci] First attempt to load modules from different compilers in a…

5d22dd7

…n env using one principal (preferred) compiler

climbfuji force-pushed the feature/boost_gcc4intel branch from 5b75c23 to 5d22dd7 Compare August 22, 2024 18:38

climbfuji added 10 commits August 22, 2024 13:48

Add [email protected] to list of compilers in .github/workflows/ubuntu-ci-x8…

d72d4eb

…6_64-intel.yaml

Merge branch 'develop' of https://github.com/jcsda/spack-stack into f…

ea8a205

…eature/boost_gcc4intel

Update Narwhal site config: add correct gcc compiler to intel environ…

709060d

…ment

[skip ci]

0bb27fe

Switch to no cray wrappers on Narwhal for Intel

edabc30

Merge branch 'develop' of https://github.com/jcsda/spack-stack into f…

762721d

…eature/boost_gcc4intel

Update configs/sites/**: for Intel, define gcc backend compiler in co…

433f3f4

…mpilers.yaml if not already there and list in packages_intel.yaml (or packages.yaml if only an Intel build exists). Still todo: atlantis, gaea-c5, gaea-c6 (added notes)

Add missing newlines at end of file in various config files

7091de5

Half way through cleaning up spack-ext/lib/jcsda-emc/spack-stack/stac…

c31c506

…k/meta_modules.py

Move other CI workflows back in place and add gcc to compiler list in…

c01018b

… .github/workflows/ubuntu-ci-x86_64-oneapi.yaml

climbfuji mentioned this pull request Aug 23, 2024

Add github CLI to global-workflow-env #1270

Open

3 tasks

climbfuji added 2 commits August 23, 2024 08:38

Trigger CI

d13707a

Clean up changes in spack-ext/lib/jcsda-emc/spack-stack/stack/meta_mo…

99aa58d

…dules.py

climbfuji force-pushed the feature/boost_gcc4intel branch from 8ac23f7 to 99aa58d Compare August 23, 2024 14:58

climbfuji commented Aug 23, 2024

View reviewed changes

Fix compiler sorting/looping logic in spack-ext/lib/jcsda-emc/spack-s…

871fd3d

…tack/stack/meta_modules.py

climbfuji force-pushed the feature/boost_gcc4intel branch from ded8f03 to 871fd3d Compare August 23, 2024 16:17

climbfuji marked this pull request as ready for review August 23, 2024 18:09

climbfuji requested review from AlexanderRichert-NOAA, ashley314, eap and RatkoVasic-NOAA August 23, 2024 18:09

Fix unit tests for setup-meta-modules: define compiler

5e27a09

climbfuji mentioned this pull request Aug 23, 2024

Rewrite setup-meta-modules extension #1274

Open

RatkoVasic-NOAA reviewed Aug 25, 2024

View reviewed changes

RatkoVasic-NOAA approved these changes Aug 25, 2024

View reviewed changes

Merge branch 'develop' of https://github.com/jcsda/spack-stack into f…

46bb80f

…eature/boost_gcc4intel

climbfuji self-assigned this Aug 27, 2024

Merge branch 'develop' into feature/boost_gcc4intel

5706f2e

AlexanderRichert-NOAA approved these changes Aug 27, 2024

View reviewed changes

climbfuji requested a review from srherbener August 27, 2024 20:44

climbfuji added 5 commits August 27, 2024 15:43

Update configs/sites/tier1/s4

f8b393f

Bug fix in ./configs/common/packages.yaml. Specify ecflow requirement…

75f6a1e

…s correctly

Merge branch 'develop' into feature/boost_gcc4intel

bfd765e

Update configs/sites/tier1/narwhal and doc/source/PreConfiguredSites.…

7b3c9d3

…rst for UCX bug fixes

Merge branch 'feature/boost_gcc4intel' of https://github.com/climbfuj…

57b83b5

…i/spack-stack into feature/boost_gcc4intel

climbfuji mentioned this pull request Aug 28, 2024

Update Atlantis and Blackpearl site configs #1277

Merged

5 tasks

srherbener reviewed Aug 28, 2024

View reviewed changes

srherbener approved these changes Aug 28, 2024

View reviewed changes

climbfuji merged commit bcd873d into JCSDA:develop Aug 28, 2024
8 checks passed

climbfuji deleted the feature/boost_gcc4intel branch August 28, 2024 15:25

climbfuji mentioned this pull request Aug 30, 2024

spack-stack-1.8.0 release tasks #1278

Open

72 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable modules for packages not compiled with preferred compiler (update spack-stack setup-meta-modules); remove need for external `ecflow` #1257

Enable modules for packages not compiled with preferred compiler (update spack-stack setup-meta-modules); remove need for external `ecflow` #1257

climbfuji commented Aug 20, 2024 •

edited

Loading

climbfuji Aug 23, 2024

RatkoVasic-NOAA left a comment

ashley314 commented Aug 27, 2024

climbfuji commented Aug 27, 2024

AlexanderRichert-NOAA left a comment

climbfuji commented Aug 27, 2024

ashley314 commented Aug 27, 2024

climbfuji commented Aug 27, 2024

srherbener Aug 28, 2024

climbfuji Aug 28, 2024

srherbener Aug 28, 2024

srherbener Aug 28, 2024

Enable modules for packages not compiled with preferred compiler (update spack-stack setup-meta-modules); remove need for external ecflow #1257

Enable modules for packages not compiled with preferred compiler (update spack-stack setup-meta-modules); remove need for external ecflow #1257

Conversation

climbfuji commented Aug 20, 2024 • edited Loading

Summary

Testing

Applications affected

Systems affected

Dependencies

Issue(s) addressed

Checklist

climbfuji Aug 23, 2024

Choose a reason for hiding this comment

RatkoVasic-NOAA left a comment

Choose a reason for hiding this comment

ashley314 commented Aug 27, 2024

climbfuji commented Aug 27, 2024

AlexanderRichert-NOAA left a comment

Choose a reason for hiding this comment

climbfuji commented Aug 27, 2024

ashley314 commented Aug 27, 2024

climbfuji commented Aug 27, 2024

srherbener Aug 28, 2024

Choose a reason for hiding this comment

climbfuji Aug 28, 2024

Choose a reason for hiding this comment

srherbener Aug 28, 2024

Choose a reason for hiding this comment

srherbener Aug 28, 2024

Choose a reason for hiding this comment

Enable modules for packages not compiled with preferred compiler (update spack-stack setup-meta-modules); remove need for external `ecflow` #1257

Enable modules for packages not compiled with preferred compiler (update spack-stack setup-meta-modules); remove need for external `ecflow` #1257

climbfuji commented Aug 20, 2024 •

edited

Loading