Implement dev -> staging -> prod

[erkannt]

Requires #787, #1067

DoD:

• prod env needs to match staging as closely as possible
• only update prod if staging is healthy and at target version (k8s job? flagger?)
• run acceptance tests as part of staging deploy
• handle failed acceptance tests (keep in failed state for analysis/rollback)
• run smoketests on production deploy

Informs #1042

ToDo:

• research approach >>> autoupdate staging with flux, run tests on prod blue/green using helm and flagger
• add browsertests to chart with helm testing hook
• install flagger
• setup canary for production (currently only for canary)
• setup canaries for submission and adaptor
• setup alerting
• docs on adding/editing canary
• docs on status/debugging canary

[erkannt]

Weavework pattern for going from stg to prod is to use istio for for canary deployments.

Their kubediff tool might allow for a simpler approach. For example it already includes a compare-images command.

I wonder if the check that staging is green and matches the desired prod image versions is best run as part of the deploy or as part of a PR pipeline that somehow queries the cluster.

Or do we move from stg to prod release via a retag of a somehow confirmed stg image.

[erkannt]

It appears that Flagger doesn't require a service mesh. It can also be used with nginx-ingress. We could run smoketests via helm hook and read only load tests.

Not sure what the best way to assert a successful test in the staging env:

• https://docs.flagger.app/usage/webhooks#integration-testing
• manually release to prod via PR
• expose a health and image tag via some form of web endpoint
• use flagger/loadtester to perform helm/grep magic to compare stg and prod

[erkannt]

Waiting for staging to be fixed.

[erkannt]

Closing this and after spinning last two todos into their own issues: #1190 #1191