Improve release management

[xiamaz]

Is your feature request related to a problem? Please describe.
Currently we have already improved the VarFish release process by having fixed maintenance windows on the Berlin instance. Still the current process lacks proper checks to avoid issues.

Describe the solution you'd like
Properly document the release process including user testing.

Ensure documented process will be able to scale to additional sites deploying VarFish.

Describe alternatives you've considered
Our current process causes frequent outages and testing in production, which works for now, but users expect more stability in the long term.

Additional context
...

[xiamaz]

Release Management proposal

varfish will be rolling release, this means primarily that varfish-server will not continue having versioned releases, but instead weekly releases, which will go through stages of testing before release.

Process

1. Devs and PM decide in dev meeting that a weekly release can be prepared. A specified commit will be tagged with a incrementing version number (no major/minor).
2. The tagged version will be rolled out to varfish-staging
3. Alpha-testers will be informed. All defined tasks need to be run by all alpha-testers for release to proceed.
4. If all tests are successful. The release will be scheduled for release on the nearest maintenance window.

Out-of-order release

In case of URGENT or P0 issues, releases can be triggered on an expediated path. Such releases only require brief acknowledgement from involved clinical party.

Out-of-order releases will require a post-mortem with analysis of causes and potential work-arounds (additional tests, process issues).

In general out-of-order releases will need to be kept to a minimum. These should also be tagged with an identical release number.

Information process

After releases are scheduled for release. Site users and external administrators will need to be informed of the new changes.

Devs and PM are responsible for collecting work that has been done for a certain tag. The tagged release should contain that information and information on upgrade impact (db rebuild etc)

PM is responsible for communicating the release info into currently used information channels, which is MS Teams. External administrators will be informed via Email including information on how an upgrade can be performed (whether any additional database work will need to be done).

Long-term, communication can move to the VarFish website.

[xiamaz]

@holtgrewe @stolpeo @einicke Need opinion if this process is okay or if you see any issues.

Currently roles as defined here are filled as follows:

Devs - Manuel, Oliver
PM - Lara, Max
Alpha testers - Lara, Max

We will need to see how much work everything is and we potentially need to shift the process around a bit. Goal is to have as little process as possible, while ensuring we can increase stability.