Parallelise daily jobs for different deployment types

[ryanemerson]

Description

Currently we create two ROSA clusters, then execute the Functional & Benchmarking tests against them for both Active/Passive, then Active/Active. The idea is that once the cluster has come up, it's then possible for members of the team to use these gh-keycloak-* clusters before they are torn down at the end of the working day in the us timezone.

However, it appears what often happens is that members of the team provision their own clusters for their work and the clusters sit idle most of the time.

My proposal is that we keep a daily scheduled run so that we can protect against regressions, however we delete the clusters as soon as the GH action run completes successfully. In the event of a test/setup failure, we should keep the cluster in order to allow a team member to investigate the root cause.

Benefits:

• Reduced costs as the ROSA clusters will only be provisioned for the duration of the tests/benchmarks, not all day.
• Increased parallelism. We can create clusters per deployment type, e.g. Active/Passive or Active/Active. This also provides increased isolation between the two environments, as we don't have to worry about state cleanup between deployment types.

Drawbacks:

• Team members will always have to explicitly provision their own clusters for dev work

Discussion

However, it appears what often happens is that members of the team provision their own clusters for their work and the clusters sit idle most of the time.

Is this a valid observation and would team members be happy to modify their existing working patterns?