Support for multi-GPU nodes (3: orchestrate different madevent executables to target different GPUs)

[valassi]

As discussed at the recent CMS meeting. This is part 3 of possible multi-GPU development.

Support for multi-GPU nodes (3: orchestrate different madevent executables to target different GPUs)

Note:

• part 1 in Support for multi-GPU nodes (1: choose one of many available GPUs) #836 is simply about choosing a single GPU (should be easy, and useful)
• part 2 in Support for multi-GPU nodes (2: send MEs to more than one GPU from the same madevent executable?) #989 is about sending MEs to more than one GPU from the same executable (complex, and not necessarily useful?)
• part 3 in Support for multi-GPU nodes (3: orchestrate different madevent executables to target different GPUs) #990 is about orchestrating from python/bash the different madevent executables to target different GPUs (not to ocomplex, and sounds very useful?)

This part 3 (which needs part 1) seems to be the most interesting of the lot... not terribly difficult: it may be enough to iterate round-robin through 4 GPUs on the system when sending jobs.

This only makes sense for workflows/processes where MEs dominate, ie very complex processes.