Add limits for simultaneous test executions

[olensmar]

To help me ensure that I'm not overloading my cluster I would like to be able to configure

• max number of total simultaneous test executions
• max number of simultaneous executions per test

When any of those limits are met - the Run button in the UI should be disabled with a corresponding warning message - "simultaneous test execution limit of XX reached" (or something like that)

[TheBrunoLopes]

I would prefer to queue tests if there are not available resources to execute them.
I'm not a big fan of adding any of the limits mentioned above

[olensmar]

sure - either could be possible/configurable - let the user decide

[ahmedimf1]

I would like to see this feature as well (or a queue as Bruno suggests), it is important for me to limit the number of test pods running concurrently to avoid resource exhaustion.

[ahmedimf1]

Perhaps something can be done with resource quotas:

https://kubernetes.io/docs/concepts/policy/resource-quotas/

Not sure if it is possible to communicate the quota error through the testkube dashboard.

[TheBrunoLopes]

Action plan:

• We Implement global value for Testkube api.
• Test / Suite CRDs should also have it.
• The lower of the two above two values is what is chosen as the value of maximum parallel tests being run.

[rangoo94]

@TheBrunoLopes, would we really like to have it only globally? Even we are encountering problems now because of the lack of the concurrency configuration on the test & test suite level.

[TheBrunoLopes]

yeah, we can also have it at the Test / Suite level.

[TheBrunoLopes]

@exu @rangoo94 @jmorante-ks to review architectural changes.

[vsukhin]

duplicated #5463