RFC: Cloud Native Deployment

[hairmare]

This RFC aims at describing the requirements needed to run Caluma in a "cloud native" way, aka on Kubernetes.

The aim of the RFC is not to invent something new, but rather to document proper guidelines on how to build Caluma for and run it on modern container orchestrators. It is assumed that this work will carry over to other solutions, like the existing docker stack or other runtimes like Podman, etc.

If implemented we would...

• Establish runtime environments based on common shared base images
• Adapt the build environment to ensure that all relevant artifacts are generated and published to appropriate source
• Implement tests, automation, and policy to demonstrate adherence to relevant standards

This would help us reach the following goals

• Caluma base images help lifecycle the os and runtime libraries and reduce the overall SBOM of a complete Caluma installation
• By adding deployment and security related artifacts to the delivery pipelines we ensure they are lifecycled in sync with their actual code, this includes artifacts like annotated SBOM data and Helm charts

In this RFC there are a few standard solutions from the CNCF ecosystem and elsewhere that are to be adopted.

Base Images

As base image, this RFC proposes adopting what the author would describe as a modern UBI stack based on UBI9 minimal. As a runtime stack, it would only be used for building packages where this is relevant to the final results (e.g. wheels).

Proposed is the following tree of images.

• ghcr.io/projectcaluma/base - this is where we gate the upstream os version and it's the root of all caluma images
  • ghcr.io/projectcaluma/build - contains build tools for tasks like building wheels and using npm/yarn/pnpm/...
  • ghcr.io/projectcaluma/python - python runtime image that contains common dependencies for all services
    • ghcr.io/projectcaluma/caluma - existing Caluma image, but new FROM
    • ghcr.io/projectcaluma/emeis - existing Emeis image, same
    • ghcr.io/projectcaluma/alexandria
    • ... - more services, more caluma, more better
  • ghcr.io/projectcaluma/nginx - minimal dnf-less nginx image specifically for ember and similar
    • ghcr.io/projectcaluma/caluma-portal-demo - example frontend
    • ... - more frontends like potentially for alexandria and emeis
  • (ghcr.io/projectcaluma/debug) - small image with debugger for in-situ cases
  • (ghcr.io/projectcaluma/utils) - like build but with IDE tooling for web IDEs

The images in () aren't in scope of this RFC but can implemented if needed.

cosign, SBOM, the "Sec" bits of DevSecOps

We use cosign from sigstore to sign all OCI content. This allows end users to verify that container images were built on a github action worker and based on a a specific pipeline and to ensure that they were not tampered with in transit. Similar to Lets Encrypt's transparency log, it provides and independent ledger of signatures for further validation purposes with rekor.

Implementing sounds complex enough to warrant an RFC of it's own, luckily everything we need is provided by the community in a well polished manner. This includes running simple non intrusive security scans with Trivy that help catch obvious defects, this SBOM will also be provided to cosign, so it's results can independently be verified.

Helm

Helm should be well known and loved already, this RFC whats to modernise how we treat Helm charts. The news are.

• We host Helm charts in each repo and publish them to the projects OCI registry
• The main charts.caluma.io repo gets refactored and only contains a single umbrella chart

By hosting individual Helm charts for Caluma subprojects and Caluma in their respective repos we can simplify the lifecycle of the charts and detect breakage earlier. An additional benefit of this setup can also be that we will have deployed versions of individual services available at build time where they could be used for end to end testing.

The umbrella chart in the now otherwise empty projectcaluma/charts repository would also be published as OCI artifact. For the time being we would keep the current index.yaml based setup for backwards compatibility reasons as it can still be a sensible choice for the main chart in many environments.

From an orga pov, this would look as follows:

├── charts
│   └── charts
│       └── caluma/...
├── caluma
│   └── charts
│       └── caluma/...
├── alexandria
│   └── charts
│       └── alexandria/...
├── emeis
│   └── charts
│       └── emeis/...
├── caluma-portal-demo
│   └── charts
│     └── caluma-portal-demo/...

Documentation

Implementing this RFC should document a few key points in the Caluma documentation

• How to deploy Caluma on Kubernetes (and how to design the deployment architecture)
• How to verify the published artifacts and where to get information like SBOMs
• How to add additional components that build on the images
• And, last but not least, why?

Prior Art

The base image and cosign stack, works really well in production at Caluma's fav radio station where i built it years ago and i've used it for a major distro upgrade.

There is also projectcaluma/alexandria#442 which I just drafted in favor of this approach. A recently merged example of doing one part of what the alexandria PR wanted to do is in https://github.com/adfinis/Outdated/.

Examples

Put in user stories:

• As a Kubernetes user i want to deploy a caluma demo in minutes by following the happy path so i can make my boss happy
• As the persons boss i want to use a demo to convince the enterprise to adopt Caluma and i need to cover stake holders like security and my boss

Specifically, this should just work ™️

helm repo add caluma https://charts.caluma.io
helm install caluma caluma/caluma

Caluma's users should be able to verify the the artifacts:

$ cosign verify ghcr.io/projectcaluma/caluma \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  --certificate-identity-regexp='https://github.com/projectcaluma/caluma/.*'

Verification for ghcr.io/projectcaluma/caluma:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

# ... JSON output here

Conclusion

This RFC proposes that we do some shift-lefting soon hoping that this is aligned with future needs where users often prefer installing an application on Kubernetes if it is well packaged for that purpose.. Done properly, it should help drive Caluma adoption in critical sectors.

The resulting code of this proposal doesn't need a name, given that it it aims to introduce common best practices and infra should vanish once working :)

I'll try to amend this RFC with some additional links to relevant concepts that i forgot to add over the next few days: openssf, slsa, oci, owasp DevSecOps Guidelines. feedback is more than welcome in any case.

[hairmare]

since creating this RFC, i have also written up an internal proposal @adfinis that could serve as the reference implementation for the user side of this solution, including the reporting and SIEM bits that are often required in regulated industries

if need be i can gladly provide more details, but most of it is out of scope for this RFC. i guess, ping me for more info if you have similar reqs...