Not sure storage flags are being honoured

[bschroeter]

Hey @SeanBryan51,

In trying to run integration tests for #143, with a pre-canned configuration file it does not appear that the flux site storage flags are actually being set.

benchcab/benchcab/utils/pbs.py

Lines 32 to 35 in 0b36493

	storage_flags = [
	"gdata/ks32",
	"gdata/hh5",
	*pbs_config.get("storage", internal.FLUXSITE_DEFAULT_PBS["storage"]),

My config file (rendered at runtime):

project: rp23

experiment: AU-Tum

realisations:
  - path: trunk
  - path: branches/Users/sb8430/test-branch

modules: [
  intel-compiler/2021.1.1,
  netcdf/4.7.4,
  openmpi/4.1.0
]

fluxsite:
    pbs:
        storage:
            - scratch/rp23

What eventuates is a job trying to run without knowledge of the rp23 scratch mount point - meaning that integration tests on my local dev installation of benchcab on rp23 simply fail with a file not found.

Am I doing something wrong, or is there an issue with passing the user-supplied flux site pbs directives through to the job script renderer?

[SeanBryan51]

Thanks for finding this. It looks like a bug that was in the main branch. Running benchcab with pdb shows we are not passing in the pbs config correctly:

benchcab/benchcab/benchcab.py

Line 137 in 7ef9fff

pbs_config=self.config.get("pbs"),

The above line will always be None since the pbs key is nested under the fluxsite key.