Use intermediate stage to avoid invalidating Docker cache when copyin…

…g from build context (#3179)

## Description

In WATonomous/infra-config#3176, we documented a
manual procedure to fix cache invalidation issues for the provisioner
container. However, there's a workaround: use a lightweight intermediate
stage to serve as the courier between the context. The workaround comes
from
[here](devcontainers/cli#153 (comment)).
This PR implements the workaround and updates the docs to reflect this.

**How it works**: Docker computes hashes for the input to each layer to
determine whether that layer can use the cache. Previously, the
permission bits on the computer that built the cache and my personal
environment were different. This resulted in the hash being different,
thus invalidating the cache. `COPY --chmod` doesn't help either because
the hash is computed [without taking into account the
parameters](docker/buildx#1311). In this PR,
we implement a workaround, where we use an extremely lightweight stage
(`FROM scratch` with only `COPY` statements) to import files and set
permissions from the build context. We don't care whether this stage
runs or gets cached, because it's very lightweight. At build-time, if we
are lucky that our system has the same permissions as the cache build
environment, then this stage will be cached. If not, this stage will
run. Regardless, the output of this stage will remain the same if the
file is unchanged. This property allows Docker to continue subsequent
steps with cache.

Tested to work with the existing cache when using permission 644. This
PR changes the permissions to 400 to be a bit more strict.

Resolves WATonomous/infra-config#3178

## Checklist
- [x] I have read and understood the [WATcloud
Guidelines](https://cloud.watonomous.ca/docs/community-docs/watcloud/guidelines)
- [x] I have performed a self-review of my code

pages/docs/community-docs/watcloud/development-manual.mdx

@@ -156,26 +156,21 @@ The cache is used to speed up both [CI](https://github.com/WATonomous/infra-conf
 and [local development](https://github.com/WATonomous/infra-config/pull/3175).
 
 The cache is automatically used. Without any changes, the following command should complete quickly[^build-time-with-cache]
-and show that every stage is loaded from cache:
+and show that almost every stage is loaded from cache:
 
 ```bash copy
 docker compose build provisioner
 ```
 
-On some setups, `COPY` and `ADD` commands may not be cached.
-For example, when there is a custom `umask` set when git checks out files, the permissions of the files may not match the cache.
-This is due to how git and Docker handle file permissions differently[^git-docker-permission-difference].
-To fix this, we can run the following to reset the non-executable bits of the files to the default:
-
-```bash copy
-git ls-files | xargs -I '{}' chmod u+rw,go+r-w {}
-```
+Previously, there was a [cache invalidation issue](https://github.com/WATonomous/infra-config/pull/3176) when the files in
+the Docker context don't have the same permissions as the cache[^git-docker-permission-difference].
+However, this issue has been fixed using a [workaround](https://github.com/WATonomous/infra-config/pull/3179).
 
 [^caching-details]: [Here](https://github.com/WATonomous/infra-config/blob/121af9af1dbe78e187670163545fa6537a26757f/.github/workflows/push-images.yml#L62) is where we push the cache, and [here](https://github.com/WATonomous/infra-config/blob/121af9af1dbe78e187670163545fa6537a26757f/docker-compose.yml#L5-L6) is where we use it. The cache lives [here](https://github.com/WATonomous/infra-config/pkgs/container/infra-config).
 
 [^build-time-with-cache]: At the time of writing (2024-09-16), the build time with cache is about 30 seconds on a single core (Docker immediately recognizes that every layer can be cached, and downloads the image from the cache). The build time without cache is about 3 minutes and 40 seconds on 8 cores.
 
-[^git-docker-permission-difference]: Git [only preserves the executable bit](https://stackoverflow.com/a/3211396/4527337) of files, and uses the umask ([defaults to `022`](https://man7.org/linux/man-pages/man2/umask.2.html) on most systems) to determine the permissions of the files it creates. Docker, on the other hand, [uses all permission bits](https://docs.docker.com/engine/reference/builder/#copy) when using `COPY` or `ADD`.
+[^git-docker-permission-difference]: Git and Docker handle file permissions differently. Git [only preserves the executable bit](https://stackoverflow.com/a/3211396/4527337) of files, and uses the umask ([defaults to `022`](https://man7.org/linux/man-pages/man2/umask.2.html) on most systems) to determine the permissions of the files it creates. Docker, on the other hand, [uses all permission bits](https://docs.docker.com/engine/reference/builder/#copy) when using `COPY` or `ADD`.
 
 
 ### Port forwarding