flux-module: load sched-fluxion-resource: module not found in search path

[christopherwharrop-noaa]

I am working on getting Flux going in a new Google Compute Platform Slurm cluster (Rocky 8 OS). I've built Flux (core 0.58.0 and sched 0.32.0) with Spack. When I try to start flux I get:

$ which flux
~/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-core-0.58.0-ixdubhz/bin/flux
$ flux start -v
May 13 17:47:12.624528 broker.err[0]: rc1.0: flux-module: load sched-fluxion-resource: module not found in search path
May 13 17:47:12.625155 broker.err[0]: rc1.0: /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-core-0.58.0-ixdubhz/etc/flux/rc1 Exited (rc=1) 0.4s

Can someone please help me find out why the module is not found? I've built flux in Ubuntu container many times, and used in other environments and never had an issue. I must have something wrong somewhere, but not sure what it might be.

$ echo $FLUX_MODULE_PATH
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-core-0.58.0-ixdubhz/lib/flux/modules
$ ls $FLUX_MODULE_PATH
barrier.so  connector-local.so  content-files.so  content.so  content-sqlite.so  cron.so  heartbeat.so  job-archive.so  job-exec.so  job-info.so  job-ingest.so  job-list.so  job-manager.so  kvs.so  kvs-watch.so  resource.so  sched-simple.so  sdbus.so  sdexec.so

[garlick]

Something needs to add flux-sched's module directory to FLUX_MODULE_PATH.

Does spack load flux-sched need to be run?

[christopherwharrop-noaa]

The problem is that the directory doesn't exist.

$ module show flux-sched
-------------------------------------------------------------------
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/modulefiles/gcc/11.2.1/flux-sched/0.32.0:

module-whatis   {A scheduler for flux-core}
module          load boost/1.83.0
module          load flux-core/0.58.0
module          load py-jsonschema/4.17.3
module          load py-pyyaml/6.0
module          load util-linux-uuid/2.38.1
module          load yaml-cpp/0.6.3
prepend-path    LD_LIBRARY_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib
prepend-path    DYLD_LIBRARY_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib
prepend-path    LD_LIBRARY_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib64
prepend-path    DYLD_LIBRARY_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib64
prepend-path    CMAKE_PREFIX_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/.
prepend-path    CMAKE_PREFIX_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/.
prepend-path    --delim ; LUA_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/share/lua/5.3/?.lua
prepend-path    --delim ; LUA_CPATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/lua/5.3/?.so
prepend-path    FLUX_MODULE_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux/modules
prepend-path    FLUX_MODULE_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux/modules/sched
prepend-path    FLUX_EXEC_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/libexec/flux/cmd
prepend-path    FLUX_RC_EXTRA /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/etc/flux
setenv          flux_sched_ROOT /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n
-------------------------------------------------------------------
$ ls /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux/modules/sched
ls: cannot access '/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux/modules/sched': No such file or directory
$ 

[christopherwharrop-noaa]

I'm sorry. I forgot that I had manually manipulated FLUX_MODULE_PATH to remove non-existent directories while trying to debug. The real path that is getting set is:

FLUX_MODULE_PATH=/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux/modules/sched:/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux
/modules:/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-core-0.58.0-ixdubhz/lib/flux/modules

But only the path I quoted above exists. Something must not be getting installed properly?

[christopherwharrop-noaa]

$ ls /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux/modules
ls: cannot access '/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux/modules': No such file or directory
$ ls /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux/modules/sched
ls: cannot access '/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/flux/modules/sched': No such file or directory
$ ls /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-core-0.58.0-ixdubhz/lib/flux/modules
barrier.so          content-files.so  content-sqlite.so  heartbeat.so    job-exec.so  job-ingest.so  job-manager.so  kvs-watch.so  sched-simple.so  sdexec.so
connector-local.so  content.so        cron.so            job-archive.so  job-info.so  job-list.so    kvs.so          resource.so   sdbus.so
 

[garlick]

Huh, yeah, the modules should be in .../flux-sched-0.32.0-ozrln6n/lib/flux/modules I believe. So spack install flux-sched completed without any errors?

[christopherwharrop-noaa]

Yes. The build succeeded. I remember that there was at one time an issue when having openmpi built with +static. Would that have anything to do with it? Now that openmpi is no longer a Spack dependency, I haven't paid as much attention to it.

[garlick]

Quick thought - possibly a naive one given I don't know that much about spack. When you were working with @trws on #1198, is it possible an unpopulated flux-sched root was created and you are loading that one instead of the successfully built one?

[christopherwharrop-noaa]

A lot of stuff got installed in that directory. Most of it is in a .spack/repos/builtin/packages subdir. I've included the files NOT in that dir here:

[Christopher.W.Harrop@chiltepindevamd-25 ~]$ find /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n -type f | grep -v '\.spack'
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/etc/flux/rc1.d/01-sched-fluxion
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/etc/flux/rc3.d/01-sched-fluxion
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/python3.10/site-packages/fluxion/jsongraph/objects/__init__.py
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/python3.10/site-packages/fluxion/jsongraph/objects/edge.py
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/python3.10/site-packages/fluxion/jsongraph/objects/graph.py
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/python3.10/site-packages/fluxion/jsongraph/objects/multigraph.py
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/python3.10/site-packages/fluxion/jsongraph/objects/node.py
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/python3.10/site-packages/fluxion/resourcegraph/V1.py
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/lib/python3.10/site-packages/fluxion/resourcegraph/__init__.py
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/libexec/flux/cmd/flux-ion-R.py
/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-0.32.0-ozrln6n/libexec/flux/cmd/flux-ion-resource.py
[Christopher.W.Harrop@chiltepindevamd-25 ~]$ 

[trws]

That does not look right, there should be two modules as well. I'm building the package right now to see if perhaps the install or something in the spack package has gotten broken.

[trws]

I got the same thing, I'm not sure how, but the installation is broken:

❯ tree
.
├── etc
│   └── flux
│       ├── rc1.d
│       │   └── 01-sched-fluxion
│       └── rc3.d
│           └── 01-sched-fluxion
├── lib
│   └── python3.11
│       └── site-packages
│           └── fluxion
│               ├── jsongraph
│               │   └── objects
│               │       ├── edge.py
│               │       ├── graph.py
│               │       ├── __init__.py
│               │       ├── multigraph.py
│               │       └── node.py
│               └── resourcegraph
│                   ├── __init__.py
│                   └── V1.py
├── lib64
│   └── flux
│       └── python3.11
│           └── fluxion -> /Users/scogland1/Workspace/dir.noindex/Repos/spack/opt/spack/linux-fedora40-m1/gcc-14.0.1/flux-sched-0.33.1-a4ezv4fd2cpcbtjzrxtxcgx5m3g57cuf/lib/python3.11/site-packages/fluxion
└── libexec
    └── flux
        └── cmd
            ├── flux-ion-resource.py
            └── flux-ion-R.py

[trws]

Ugh, I think I see what's happening. It's installing to FLUX_MOD_DIR, which is where flux-core installed its modules, rather than the same subdirectory under the install prefix. It normally happens to work because flux-core's install prefix will be writable in spack, and will be a valid module dir, but it means the combination of the flux-sched build and the spack package definitely doesn't do as expected.

[christopherwharrop-noaa]

I think I'll try removing flux-sched from my spack binary buildcache and reinstall it to see if that changes anything.

[christopherwharrop-noaa]

I just realized I probably can't remove the package from the spack build cache at the moment because of a temporary authentication issue (totally unrelated) I'm having with S3 (where my buildcache is located) that may take some time to resolve.

[christopherwharrop-noaa]

@trws - Thank you for the update! Glad I wasn't hallucinating. Let me know if there's anything I can do to test/try.

[christopherwharrop-noaa]

It's trying to fetch from this repo instead of from @trws 's fork. Do you know how to tell Spack to fetch from the fork?

==> [2024-05-13-23:39:25.935377] Fetching from s3://chiltepin-us-east-2/spack-stack/_source-cache/git/flux-framework/flux-sched.git/6942db48da9991ecaab05cfb7f88064b725e0514.tar.gz failed.
==> [2024-05-13-23:39:25.935412] Fetching from s3://chiltepin-us-east-2/spack-stack/flux-sched/flux-sched-git.6942db48da9991ecaab05cfb7f88064b725e0514_0.34.1.tar.gz failed.
==> [2024-05-13-23:39:25.935441] Fetching from https://mirror.spack.io/_source-cache/git/flux-framework/flux-sched.git/6942db48da9991ecaab05cfb7f88064b725e0514.tar.gz failed.
==> [2024-05-13-23:39:25.935469] Fetching from https://mirror.spack.io/flux-sched/flux-sched-git.6942db48da9991ecaab05cfb7f88064b725e0514_0.34.1.tar.gz failed.
==> [2024-05-13-23:39:25.935495] Fetching from [git] https://github.com/flux-framework/flux-sched.git at commit 6942db48da9991ecaab05cfb7f88064b725e0514 failed.
==> [2024-05-13-23:39:25.960339] Error: FetchError: All fetchers failed for spack-stage-flux-sched-git.6942db48da9991ecaab05cfb7f88064b725e0514_0.34.1-kjw5o6jbmvqw3h6mx3n2toylvyttbtar

[trws]

Ah, indeed. I'm not sure if there's a way to override that unfortunately.

[christopherwharrop-noaa]

I asked over on the Spack Slack channel. I'll try it out as soon as I can figure out how to do it. Thanks once again for your very quick response and diagnosis! I so appreciate it.

[christopherwharrop-noaa]

I hacked the flux-sched package.py file in my Spack install to point to your fork. It looked like that allowed it to find it, but unfortunately, it doesn't like the version again.

1 error found in build log:
  >> 4    CMake Error at CMakeLists.txt:15 (project):
     5      VERSION "git.6942db48da9991ecaab05cfb7f88064b725e0514=0.34.1" format
     6      invalid.
     7    
     8    
     9    -- Configuring incomplete, errors occurred!

See build log for details:

[christopherwharrop-noaa]

I made sure I had the updated package.py with the cmake fix in it, and I also tried manually setting export FLUX_SCHED_VERSION=0.34.1 and export FLUX_SCHED_VER=0.34.1 but that did not help with the version format issue. Unfortunately, I'm unable to test the PR as I can't figure out how to get Spack to build it. If you know of a way to work around this, please let me know. I'm eager to try this.

[christopherwharrop-noaa]

@trws I followed your thread on Slack about getting the package attributes set to add flux-sched from your fork for testing. And I was able to use spack config add "packages:flux-sched:package_attributes:git:'https://github.com/trws/flux-sched.git'" to set that up so that I don't have to hack the package.py file. The thing I'm struggling with now is that no matter what I do, it appears that CMake is passing -DFLUX_SCHED_VER:STRING=git.6942db48da9991ecaab05cfb7f88064b725e0514=0.34.1 and so it complains about invalid version format (as reported above). I guess it's getting that info from the Spack spec used in the spack add command? The environment variables that worked before don't seem to influence what gets passed to CMake. If you have any recommendations for how to work around this please let me know.

[christopherwharrop-noaa]

If that wasn't something you wanted to deal with, and you have a quick one-off fix for that, I could just tweak it for the purposes of installing this PR branch for testing. I'm sorry to say I'm not super savvy with Spack package files so would need a little bit of guidance to make that tweak. But I'm more than happy to do the one-off experiment to get past this issue for testing purposes.

[trws]

The fix is to swap the formatting of self.spec.version to using self.spec.version.ref_version. I didn't think about the fact that the version normally stays fully qualified when all we actually want is the effective version number.

[christopherwharrop-noaa]

Thank you. That worked to get it to build and install. And I see the modules in there now:

$ tree
.
├── etc
│   └── flux
│       ├── rc1.d
│       │   └── 01-sched-fluxion
│       └── rc3.d
│           └── 01-sched-fluxion
├── lib
│   └── python3.10
│       └── site-packages
│           └── fluxion
│               ├── jsongraph
│               │   └── objects
│               │       ├── edge.py
│               │       ├── graph.py
│               │       ├── __init__.py
│               │       ├── multigraph.py
│               │       └── node.py
│               └── resourcegraph
│                   ├── __init__.py
│                   └── V1.py
├── lib64
│   └── flux
│       ├── libreapi_cli.so
│       ├── modules
│       │   ├── sched-fluxion-qmanager.so
│       │   └── sched-fluxion-resource.so
│       └── python3.10
│           └── fluxion -> /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-git.6942db48da9991ecaab05cfb7f88064b725e0514_0.34.1-ihzr2ht/lib/python3.10/site-packages/fluxion
└── libexec
    └── flux
        └── cmd
            ├── flux-ion-resource.py
            └── flux-ion-R.py

19 directories, 14 files

I'm going to try out the install and report results, which will take me a bit of time, but wanted to report this quick win. Thanks for all your help so far.

[trws]

Glad to hear it, and thank you for helping debug and diagnose this. It's been valuable.

[christopherwharrop-noaa]

When I tried to launch flux I got that same module not found error again:

$ flux start
May 16 16:03:14.923324 broker.err[0]: rc1.0: flux-module: load sched-fluxion-resource: module not found in search path
May 16 16:03:14.923858 broker.err[0]: rc1.0: /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-core-0.58.0-ixdubhz/etc/flux/rc1 Exited (rc=1) 0.3s

That seemed wrong, so I looked at the tcl module file to make sure it is setting FLUX_MODULE_PATH correctly and found out that it is prepending lib/flux/modules but the modules are located in lib64/flux/modules:

prepend-path    FLUX_MODULE_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-git.6942db48da9991ecaab05cfb7f88064b725e0514_0.34.1-ihzr2ht/lib/flux/modules
prepend-path    FLUX_MODULE_PATH /home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-sched-git.6942db48da9991ecaab05cfb7f88064b725e0514_0.34.1-ihzr2ht/lib/flux/modules/sched

I manually corrected the FLUX_MODULE_PATH to use lib64 for flux-sched and the module problem went away and I just got:

$ flux start
'abrt-cli status' timed out

Which is likely just an issue with me trying to start it in the wrong context.

I hope this info is useful. I'm going to proceed with trying to make use of Flux in the proper environment with these fixes included to see if I can get it to behave. Let me know if you need me to try anything else.

[christopherwharrop-noaa]

Sorry for the long delay. It looks like the PR version of flux-sched does work for me. I am using it on a Google Compute Platform Slurm cluster with the Parsl FluxExecutor, so it took me a while to work through some things. I still have some tweaking to get all my tests working, but enough of them are working to prove that Flux is functional now.

[christopherwharrop-noaa]

@trws I saw you pushed an update so I checked it out and tested. I had success with it starting Flux and flux-sched modules now get installed into lib instead of lib64 . All my Flux tests that don't involve executing MPI programs pass now. My only remaining problem is that I can't get my MPI programs to run. They crash in MPI_Init():

*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[christopherwharrop-chiltepindevamd-00031-3-0001:56494] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[christopherwharrop-chiltepindevamd-00031-3-0001:56495] OPAL ERROR: Unreachable in file ext3x_client.c at line 111

I think that is almost for sure totally unrelated to the problem reported here. I noticed that Spack had installed openmpi with +static even though I told it to use ~static. I know that has caused some issues related to the launching of codes with OpenMPI running under Flux. I also know that FLUX_PMI_LIBRARY_PATH might need tweaking. I've tried:

export FLUX_PMI_LIBRARY_PATH=/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-core-0.58.0-ixdubhz/lib/flux/libpmi.so
export FLUX_PMI_LIBRARY_PATH=/home/Christopher.W.Harrop/spack-stack/envs/unified-dev.mylinux/install/gcc/11.2.1/flux-core-0.58.0-ixdubhz/lib/flux/libpmi2.so

and neither of them work. But, like I said, I think the MPI launch issue is a different problem altogether.

Anyway, thank you so much for all your work on this.

[trws]

Thanks @christopherwharrop-noaa, I'm glad to hear it. Hopefully we'll get that merged shortly and I'll see about getting the other spack package tweak pushed upstream as well.

The MPI issue does sound separate, but it could well be OpenMPI not pulling our PMI, especially if it's version 5 since that drops support for everything but PMIX. If you have to use OpenMPI, you might consider trying our pmix support by adding in the flux-pmix project. It's less battle-tested, but newer OpenMPIs really don't play nice with anything else. Alternately anything mpich-derived should work unless very carefully and specifically built to disable their default pmi interfaces. We implement their wire protocol for PMI-simple directly so that tends to be amazingly bulletproof (through vendor extensions, with other job systems enabled, in containers, it tends to "just work" in a way the others don't).

[christopherwharrop-noaa]

Thank you for the extra tips and for plans to add that tweak to the Spack package file.

Please don't let my OpenMPI woes keep you from wrapping up the PR or from closing this discussion. I have a good idea where to look to solve that one. As far as I'm concerned this install problem is solved.