dockerTools: Allow separately specifying metadata and filesystem time…

…stamps

Setting the image creation timestamp in the image metadata to a
constant date can cause problems with self-hosted container
registries, that need to e.g. prune old images.  This timestamp is
also useful for debugging.

However, it is almost never useful to set the filesystem timestamp to
a constant value.  Doing so not only causes the image to possibly no
longer be reproducible, but also removes any possibility of
deduplicating layers with other images, causing unnecessary storage
space usage.

Therefore, this commit introduces "mtime", a new parameter to
streamLayeredImage, which allows specifying the filesystem timestamps
separately from "created".  For backwards compatibility, "mtime"
defaults to the value of "created".

doc/build-helpers/images/dockertools.section.md

@@ -436,7 +436,7 @@ See [](#ex-dockerTools-streamLayeredImage-exploringlayers) to understand how the
 `streamLayeredImage` allows scripts to be run when creating the additional layer with symlinks, allowing custom behaviour to affect the final results of the image (see the documentation of the `extraCommands` and `fakeRootCommands` attributes).
 
 The resulting repository tarball will list a single image as specified by the `name` and `tag` attributes.
-By default, that image will use a static creation date (see documentation for the `created` attribute).
+By default, that image will use a static creation date (see documentation for the `created` and `mtime` attributes).
 This allows the function to produce reproducible images.
 
 ### Inputs {#ssec-pkgs-dockerTools-streamLayeredImage-inputs}
@@ -499,6 +499,7 @@ This allows the function to produce reproducible images.
 `created` (String; _optional_)
 
 : Specifies the time of creation of the generated image.
+  This date will be used for the image metadata, and as the default value for `mtime`.
   This should be either a date and time formatted according to [ISO-8601](https://en.wikipedia.org/wiki/ISO_8601) or `"now"`, in which case the current date will be used.
 
   :::{.caution}
@@ -507,6 +508,18 @@ This allows the function to produce reproducible images.
 
   _Default value:_ `"1970-01-01T00:00:01Z"`.
 
+`mtime` (String; _optional_)
+
+: Specifies the time used for the modification timestamp of files within the layers of the generated image.
+  This should be either a date and time formatted according to [ISO-8601](https://en.wikipedia.org/wiki/ISO_8601) or `"now"`, in which case the current date will be used.
+
+  :::{.caution}
+  Using a non-constant date will cause built layers to have a different hash each time, preventing deduplication.
+  Using `"now"` also means that the generated image will not be reproducible anymore (because the date will always change whenever it's built).
+  :::
+
+  _Default value:_ the same value as `created`.
+
 `uid` (Number; _optional_) []{#dockerTools-buildLayeredImage-arg-uid}
 `gid` (Number; _optional_) []{#dockerTools-buildLayeredImage-arg-gid}
 `uname` (String; _optional_) []{#dockerTools-buildLayeredImage-arg-uname}

pkgs/build-support/docker/default.nix

@@ -907,6 +907,7 @@ rec {
     , config ? { }
     , architecture ? defaultArchitecture
     , created ? "1970-01-01T00:00:01Z"
+    , mtime ? created
     , uid ? 0
     , gid ? 0
     , uname ? "root"
@@ -999,7 +1000,7 @@ rec {
 
         conf = runCommand "${baseName}-conf.json"
           {
-            inherit fromImage maxLayers created uid gid uname gname;
+            inherit fromImage maxLayers created mtime uid gid uname gname;
             imageName = lib.toLower name;
             preferLocalBuild = true;
             passthru.imageTag =
@@ -1019,10 +1020,13 @@ rec {
             imageTag="${tag}"
           ''}
 
-          # convert "created" to iso format
+          # convert "created" and "mtime" to iso format
           if [[ "$created" != "now" ]]; then
               created="$(date -Iseconds -d "$created")"
           fi
+          if [[ "$mtime" != "now" ]]; then
+              mtime="$(date -Iseconds -d "$mtime")"
+          fi
 
           paths() {
             cat $paths ${lib.concatMapStringsSep " "
@@ -1079,6 +1083,7 @@ rec {
               "customisation_layer", $customisation_layer,
               "repo_tag": $repo_tag,
               "created": $created,
+              "mtime": $mtime,
               "uid": $uid,
               "gid": $gid,
               "uname": $uname,
@@ -1090,6 +1095,7 @@ rec {
               --arg customisation_layer ${customisationLayer} \
               --arg repo_tag "$imageName:$imageTag" \
               --arg created "$created" \
+              --arg mtime "$mtime" \
               --arg uid "$uid" \
               --arg gid "$gid" \
               --arg uname "$uname" \

pkgs/build-support/docker/stream_layered_image.py

@@ -316,16 +316,21 @@ def add_bytes(tar, path, content, mtime):
     tar.addfile(ti, io.BytesIO(content))
 
 
+now = datetime.now(tz=timezone.utc)
+
+
+def parse_time(s):
+    if s == "now":
+        return now
+    return datetime.fromisoformat(s)
+
+
 def main():
     with open(sys.argv[1], "r") as f:
         conf = json.load(f)
 
-    created = (
-      datetime.now(tz=timezone.utc)
-      if conf["created"] == "now"
-      else datetime.fromisoformat(conf["created"])
-    )
-    mtime = int(created.timestamp())
+    created = parse_time(conf["created"])
+    mtime = int(parse_time(conf["mtime"]).timestamp())
     uid = int(conf["uid"])
     gid = int(conf["gid"])
     uname = conf["uname"]