Introduce pipeline versioning

[alanbchristie]

We need to support multiple pipeline implementation images. At the moment we have an implementation image (latest) and a poster image (latest). We need to support versioning of the pipelines at the base image and pipelines implementation levels.

The proposal is to: -

1. Put explicit image tags in each service descriptor
2. Ensure that the pipeline base images (i.e. RDKit) contain a base version using fields in the image name.
3. Produce pipelines images uses tags that define the implementation version
4. Post pipelines directly from source (rather than using a “poster” image)

An image reference in a service descriptor will therefore look like this:

informaticsmatters/rdkit_pipelines_2019-1:1

Where: -

• informaticsmatters/rdkit-pipelines is the base image
• 2019-1 is its base version (i.e. anything after the last “_” in the image name)
• 7 is the pipelines implementation version

Base and implementation versions

The base image version changes whenever code affecting the base image changes.

The implementation version changes whenever a pipeline (or the utilities it uses) changes or when the base image changes. For example if pipeline image informaticsmatters/rdkit_pipelines_2019-1:1 is updated with a new informaticsmatters/rdkit_pipelines_2019-2 base image, the new pipeline image becomes informaticsmatters/rdkit_pipelines_2019-2:2.

Pipelines in a released image can refer to different implementation versions. For example, it’s perfectly reasonable to have service descriptors referring to many different implementation or base image versions.

Pipeline images, once released (to Docker Hub for example) must not be removed, any previously released pipeline may already be in use at a customer site.

Upon release (from the master branch of a pipelines repository) the master branch must be tagged with the implementation version using the format rN where N is the release version (an integer).

Pipeline versions

Pipeline implementation version numbers increase but not necessarily in an unbroken sequence.

As pipelines often share a base image they must also share the implementation tag. It is quite possible that the implementation has not changed despite a number of new pipelines image releases. A pipeline may refer to a “old” image version for a considerable time. When the pipeline is eventually updated the implementation tag it refers to may be significantly larger than its existing version. Implementation numbers used for a given pipeline service descriptor can therefore change from 1 to 64 in a single release.

## Posting pipelines

This was previously accomplished using a “poster” image. For this feature we will “post” pipelines to a squonk deployment (via the existing REST service) directly from a check-out of a pipelines source repository - specifically from source cloned from a repository tag (i.e. r17).