Support concurrent resource usage limit >1

[mthuurne]

Currently we only support the two extremes of resource usage limits: either a resource is exclusive and it can be used by at most one task, or it is non-exclusive and it can be used by an unlimited number of tasks. It would be useful to set a limit higher than 1 as well.

For Task Runners, with the number of CPU cores growing higher and higher, the number of TRs you can run per Factory PC becomes ever larger. But having to configure for example 10 nearly-identical TRs is a chore. It would be nice if instead the operator could set the usage limit for that Task Runner to 10.

Other resource types might also benefit from this feature. For example a server might support dynamically starting multiple instances of services, but not have enough memory to support peak demand. Or if a limited-instances tool license is modeled as a resource, the number of available instances might be more than one.

Note that until now I only wrote about limits on concurrent tasks using the resource. What about per-job exclusivity? That is typically used when state has to be preserved between tasks. In the example of a server which support a limited number of service instances, the service could be started by one task and then used by multiple tasks. A product locator could be used to tell those other tasks how to connect to the right instance. So having a limit >1 for per-job exclusivity makes sense.

Should there be a single limit with per-job and per-task flags, or a separate limit for per-job and per-task? There is only a difference if a resource is both per-job and per-task exclusive; what kind of resources are those? Being per-job exclusive means that there is some state that is shared between tasks, while being per-task exclusive means that there is overlapping state that is modified within tasks. A service could be loaded with specific data for a particular job and then accessed by multiple tasks within that job at once, but not have the capacity to service an unlimited number of tasks concurrently. The job limit could model limited storage capacity, while the task limit could model limited RAM, CPU or network bandwidth. So I think it makes sense to have separate job and task limits.

Does it make sense to support a concurrent usage limit of 0? That would effectively suspend the resource (not literally: its status would be busy rather than suspended), since there are never enough units available to assign it to a task. A limit of 0 might be useful for an operator to force a resource to be temporarily offline without running the risk that a user resumes it. I'm not sure if that is enough of a reason to support it though. I'm also not sure whether implementing a limit of 0 would force a cleaner design upon the code or add complexity by requiring special cases.

A possible future feature that builds upon this is to allow task definitions to request multiple instances of a resource. In that case, you could make very fine-grained resource reservations, for example a test requesting 50GB of storage space. However, if a task asks for 50 units of 1GB storage space, it might expect to get that from a single location instead of multiple blocks on different resources that add up to 50GB. So perhaps capacity should be modeled separately from resource instances. Capacity is probably closer to a generalization of a capability than of a resource instance.