Server-side container image builds

[AdrianoKF]

As a data scientist / ML engineer, I do not want to go through the overhead of downloading all sorts of dependencies and uploading them back to the internet as a container image, before I can sub my job for execution (since I am on a limited-bandwidth connection).

Instead, I want these container images to be built on a build server, according to the image specification given as part of my job metadata.

This approach yields additional benefits:

• Improved caching (esp. in teams), since images don't need to be built multiple times on each team members' local machine
• No architecture gap (e.g., macOS/Linux, ARM/x64) between clients and compute server
• No need for data scientists to install container tooling on their machine
• Potential for security measures on the build server (e.g., vulnerability scanning, image signing, SBOM, ...)

[AdrianoKF]

I've hacked together a Trie-based path matching class that can be used to validate Gitignore (or Dockerignore) patterns against a file path: https://gist.github.com/AdrianoKF/d5bf77f200592c2cab2b8633b85f8a97

This can serve as the starting point when building an archive of the build context locally for upload to the server.