Other software projects solving same problem

[aidanheerdegen]

Given the there are other software projects that are solving the same problem why is it necessary to create another one?

https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/jobs/runners

https://github.com/Parsl/parsl/blob/dd9150d7ac26b04eb8ff15247b1c18ce9893f79c/parsl/providers/pbspro/pbspro.py#L15

https://www.nextflow.io

https://github.com/prisms-center/pbs

https://github.com/nasa/pbs4py

https://perso.lpsm.paris/~msangnier/pbspython.html

I am particularly interested because we stalled on implementing cross-platform queue support in payu when it wasn't obvious what prior art we should use:

payu-org/payu#182

So .. should payu adopt hpcpy? Should hpcpy utilise/wrap other exisiting libraries?

[bschroeter]

I evaluated most of (but not all) these libraries prior to starting HPCpy and came to the conclusion that they fell into one of 3 categories:

1. The project was essentially defunct and was no longer maintained.
2. The project was tightly integrated to a particular HPC, thus not generalisable
3. The interface was unnecessarily convoluted to do something as simple as submit a job.

A valid argument though, it has been a while since I tried some of these so they may have matured to a usable point for me. But in the meantime I like the simplicity that HPCpy provides:

from hpcpy.client import PBSClient
job_id = client.submit("jobscript.sh", queue="express")

I'm not saying HPCpy is superior by any means, it has just simplified a lot on my end!

[aidanheerdegen]

I'm not saying HPCpy is superior by any means, it has just simplified a lot on my end!

I admit the question was a little provocative, but I am genuinely interested because I didn't check out all those other options in any depth so am interested that you have.

So ... can we utilise HPCpy in payu given payu doesn't use PBS submit scripts, but calls itself on the PBS submit command line?

[bschroeter]

@aidanheerdegen, I appreciate you keeping me on my toes!

I will reserve my opinions until I look into how payu actually works in this regard.

However, on the face of it, if all payu needs to do is submit a PBS job, then this is an option for a Python way to do so.

[aidanheerdegen]

Well .. the motivation for payu-org/payu#182 is because we wanted to add SLURM support a bit more officially.

I guess that isn't a near-term goal?

[bschroeter]

@aidanheerdegen, it can be...

The development roadmap is largely self-serving to what is needed in other projects I am working on (i.e. meorg_client, am3_ci etc.) if a more imminent need for SLURM support arises we can look at scoping it out.

[bschroeter]

@aidanheerdegen, see #1 . SLURM support is something I've wanted since inception...

[aidanheerdegen]

My question has been thoroughly answered, so thank you @bschroeter