MDBF-773 - QUAY,GHCR move / create tags for Production images

[RazvanLiviuVarzaru]

Implement dev tag feature as per MDBF-773.

Scope:

• speed up production releases.
• implement a way to test the images in the Buildbot development instance before they are ready for Production.

Description:

• BuildBot (Development) will work with images tagged with DEV_ . Switching between different registries is achieved via an environment variable.
• every push to a non-Main branch will create DEV_ tags (builds + push ghcr, quay).
• pushes to Main will trigger a manifest copy in the upstream registry (no rebuild).

Other improvements:

• build RHEL images for changes pushed to a non-Main branch. No Builds for forks / PR's.
• Release container uses workflow template.

Considerations:

• workflow dispatch supports a maximum of 10 input parameters.

[vladbogo]

Maybe consider moving the CONTAINER_REGISTRY_URL in the create_worker function and only pass the tag as a param when using addWorker

[fauust]

That's a really really nice improvement!
This PR is quite big (and I understand why), but it's difficult to be entirely sure with my review. To proceed I think that (after the small change requested), it's okay to merge to DEV and start verifying directly on the DEV infra that everything looks OK.

I don't see how this could impact production if merged only on dev branch but probably other should confirm.

[RazvanLiviuVarzaru]

That's a really really nice improvement! This PR is quite big (and I understand why), but it's difficult to be entirely sure with my review. To proceed I think that (after the small change requested), it's okay to merge to DEV and start verifying directly on the DEV infra that everything looks OK.

I don't see how this could impact production if merged only on dev branch but probably other should confirm.

@fauust
I will merge this to DEV with CI disabled and then manually run all the workflows and create the DEV _ tags.
Then a test on BB-DEV on a few builders and I will watch the workers closely to see if they are pulling the DEV tag.

As for Production impact.
The merge to main should be done with CI disabled, no need to update the tags, but I'm thinking of a few workflows to let them update to tags just to confirm the feature is working properly.

[RazvanLiviuVarzaru]

Maybe consider moving the CONTAINER_REGISTRY_URL in the create_worker function and only pass the tag as a param when using addWorker

@vladbogo so if I understand correctly, most of the time adding a worker follows the above steps:
invoke AddWorker function --> addWorker will invoke createWorker function which is defined in utils.py

And we have a small percentage of workers which are defined directly without using the addWorker function but rather appending directly to c["workers"] list .

Your suggestion is really good but I would keep this kind of refactoring for a different patch where we normalize all worker addition to follow the addWorker path. I'm a little bit concerned about adding it to this patch as there are already a lot of changes .