Support command line invocation (non-script) use

[aidanheerdegen]

We have a requirement to support SLURM with payu, but payu submits itself on the command line (doesn't use a script).

To utilise hpcpy it would need to support this use case. Would this be difficult? What would be required?

[bschroeter]

@aidanheerdegen, if I am not mistaken payu-run is a script itself?

There is nothing stopping the user from providing this absolute path to submit(). However, we would need to implement #25 in order for this to replicate the existing functionality of payu, no?

[aidanheerdegen]

if I am not mistaken payu-run is a script itself?

It's an entrypoint, but yeah, good point

https://github.com/payu-org/payu/blob/master/pyproject.toml#L47

It looks like this when invoked

$ payu run
payu: warning: Job request includes 47 unused CPUs.
payu: warning: CPU request increased from 241 to 288
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
payu: Found modules in /opt/Modules/v4.3.0
qsub -q normal -P tm70 -l walltime=18000 -l ncpus=288 -l mem=1000GB -N 1deg_jra55_ryf -l wd -j n -v PAYU_PATH=/g/data/vk83/prerelease/apps/payu/dev/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/etc/scl/modulefiles:/g/data/vk83/prerelease/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/vk83+scratch/tm70 -- /g/data/vk83/prerelease/apps/payu/dev/bin/python /g/data/vk83/prerelease/apps/payu/dev/bin/payu-run

payu-run is automatically generated, and looks like this:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re
import sys
from payu.subcommands.run_cmd import runscript
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                                      
    sys.exit(runscript())

[jo-basevi]

I think Payu still runs the qsub command as a subprocess in python. E.g. (https://github.com/payu-org/payu/blob/f4af1ffaf49381a8a789faf77d75ee19ab89fb8e/payu/cli.py#L163-L173):

def submit_job(script, config, vars=None):
    """Submit a userscript the scheduler."""

    # TODO: Temporary stub to replicate the old approach
    sched_name = config.get('scheduler', 'pbs')
    sched_type = scheduler_index[sched_name]
    sched = sched_type()
    cmd = sched.submit(script, config, vars)
    print(cmd)

    subprocess.check_call(shlex.split(cmd))

So the sched.submit() generates the qsub command. So I think using hpcpy as is could work? I'm sorry if I am missing something!

Payu qsub commands use -- to run commands rather a script e.g., qsub <directives> -- /g/data/vk83/prerelease/apps/payu/dev/bin/python /g/data/vk83/prerelease/apps/payu/dev/bin/payu-run. I am almost wondering whether payu could have Jinja templated job shell script that basically just ran {{python}} {{payu-cmd}} as this would solve some current module initialisation issues in payu (payu-org/payu#522)..

[aidanheerdegen]

I am almost wondering whether payu could have Jinja templated job shell script that basically just ran {{python}} {{payu-cmd}} as this would solve some current module initialisation issues in payu (payu-org/payu#522)..

Interesting ... I guess it could.