8.11 Docker image does not run with non-default Linux UIDs and size has increased to 4.13 GB from 2.3 GB

[cmacknz]

• Relates [Fix] Agent incapable of running on Azure Container Instances #3576

The change from #3576 has unintentionally doubled the size of our Docker image but more importantly is a breaking change for users that deploy agent with non-standard UID or GIDs. This is common in ECE deployments, if an ECE user has ECE installed with a non-standard (non-1000) UID/GID the agent and APM won't boot up.

Done preparing, starting Elastic Agent and Apm. See Elastic Agent and Apm logs for further output.
/app/apm.sh: line 89: /usr/share/elastic-agent/elastic-agent: Permission denied
Done preparing, starting Elastic Agent and Apm. See Elastic Agent and Apm logs for further output.
/app/apm.sh: line 89: /usr/share/elastic-agent/elastic-agent: Permission denied
2023-11-06T19:03:54+0000 Booting at Mon Nov 6 19:03:54 UTC 2023
Done preparing, starting Elastic Agent and Apm. See Elastic Agent and Apm logs for further output.
/app/apm.sh: line 89: /usr/share/elastic-agent/elastic-agent: Permission denied

Our Docker image size in 8.11 has increased 4.13 GB, which appears to be because of an additional Docker layer touching all files in the file system.

docker images | grep elastic-agent-cloud
docker.elastic.co/cloud-release/elastic-agent-cloud 8.11.0-e75ef729 fe8243cd3cad 45 hours ago 4.13GB
docker.elastic.co/cloud-release/elastic-agent-cloud 8.10.0          d500f2e33781  8 weeks ago 2.3GB

Using https://github.com/GoogleContainerTools/container-diff shows the actual container size has barely changed:

container-diff diff docker.elastic.co/cloud-release/elastic-agent-cloud:8.11.0-e75ef729 docker.elastic.co/cloud-release/elastic-agent-cloud:8.10.0

-----Size-----

Image size difference between docker.elastic.co/cloud-release/elastic-agent-cloud:8.11.0-e75ef729 and docker.elastic.co/cloud-release/elastic-agent-cloud:8.10.0:
SIZE1        SIZE2
2.2G         2.1G

The reason for the increase is revealed looking at the layer history:

8.11.0-e75ef729
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
fe8243cd3cad   45 hours ago   RUN /bin/sh -c echo -e '#!/bin/sh\nexec /usr…   51B       buildkit.dockerfile.v0
<missing>      45 hours ago   CMD ["/app/apm.sh"]                             0B        buildkit.dockerfile.v0
<missing>      45 hours ago   ENTRYPOINT ["/usr/bin/tini" "--"]               0B        buildkit.dockerfile.v0
<missing>      45 hours ago   WORKDIR /usr/share/elastic-agent                0B        buildkit.dockerfile.v0
<missing>      45 hours ago   ENV LIBBEAT_MONITORING_CGROUPS_HIERARCHY_OVE…   0B        buildkit.dockerfile.v0
<missing>      45 hours ago   USER elastic-agent                              0B        buildkit.dockerfile.v0
<missing>      45 hours ago   RUN /bin/sh -c setcap cap_net_raw,cap_setuid…   105MB     buildkit.dockerfile.v0
<missing>      45 hours ago   RUN /bin/sh -c chown elastic-agent /app # bu…   0B        buildkit.dockerfile.v0
<missing>      45 hours ago   RUN /bin/sh -c mkdir /app # buildkit            0B        buildkit.dockerfile.v0
<missing>      45 hours ago   RUN /bin/sh -c groupadd --gid 1000 elastic-a…   1.77GB    buildkit.dockerfile.v0
<missing>      45 hours ago   COPY /opt /opt # buildkit                       367MB     buildkit.dockerfile.v0
<missing>      45 hours ago   COPY /usr/share/elastic-agent/NOTICE.txt /li…   1.03MB    buildkit.dockerfile.v0
<missing>      45 hours ago   COPY /usr/share/elastic-agent/LICENSE.txt /l…   13.7kB    buildkit.dockerfile.v0
<missing>      45 hours ago   RUN /bin/sh -c mkdir /licenses # buildkit       0B        buildkit.dockerfile.v0
<missing>      45 hours ago   RUN /bin/sh -c chmod 0770 /usr/share/elastic…   0B        buildkit.dockerfile.v0
<missing>      45 hours ago   COPY /usr/share/elastic-agent /usr/share/ela…   1.77GB    buildkit.dockerfile.v0
<missing>      45 hours ago   RUN /bin/sh -c chmod 755 /usr/local/bin/dock…   426B      buildkit.dockerfile.v0
<missing>      45 hours ago   COPY docker-entrypoint /usr/local/bin/docker…   426B      buildkit.dockerfile.v0
<missing>      45 hours ago   RUN /bin/sh -c set -e ;   TINI_BIN="";   TIN…   23.9kB    buildkit.dockerfile.v0
<missing>      45 hours ago   ENV GODEBUG=madvdontneed=1                      0B        buildkit.dockerfile.v0
<missing>      45 hours ago   ENV PATH=/usr/share/elastic-agent:/usr/local…   0B        buildkit.dockerfile.v0
<missing>      45 hours ago   ENV ELASTIC_CONTAINER=true                      0B        buildkit.dockerfile.v0
<missing>      45 hours ago   LABEL org.label-schema.build-date=2023-11-04…   0B        buildkit.dockerfile.v0
<missing>      45 hours ago   RUN /bin/sh -c for iter in {1..10}; do      …   50.2MB    buildkit.dockerfile.v0
<missing>      45 hours ago   ENV BEAT_SETUID_AS=elastic-agent                0B        buildkit.dockerfile.v0
<missing>      4 weeks ago    /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B        
<missing>      4 weeks ago    /bin/sh -c #(nop) ADD file:f70cc2610ea8fcd25…   65.7MB    
<missing>      4 weeks ago    /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      4 weeks ago    /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      4 weeks ago    /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B        
<missing>      4 weeks ago    /bin/sh -c #(nop)  ARG RELEASE                  0B        

8.10.0
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
d500f2e33781   8 weeks ago    RUN /bin/sh -c echo -e '#!/bin/sh\nexec /usr…   51B       buildkit.dockerfile.v0
<missing>      8 weeks ago    CMD ["/app/apm.sh"]                             0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    ENTRYPOINT ["/usr/bin/tini" "--"]               0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    WORKDIR /usr/share/elastic-agent                0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    ENV LIBBEAT_MONITORING_CGROUPS_HIERARCHY_OVE…   0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    USER elastic-agent                              0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c chown elastic-agent /app # bu…   0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c mkdir /app # buildkit            0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c useradd -M --uid 1000 --gid 1…   333kB     buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c groupadd --gid 1000 elastic-a…   1.68kB    buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c setcap cap_net_raw,cap_setuid…   105MB     buildkit.dockerfile.v0
<missing>      8 weeks ago    COPY /opt /opt # buildkit                       355MB     buildkit.dockerfile.v0
<missing>      8 weeks ago    COPY /usr/share/elastic-agent/NOTICE.txt /li…   1MB       buildkit.dockerfile.v0
<missing>      8 weeks ago    COPY /usr/share/elastic-agent/LICENSE.txt /l…   13.7kB    buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c mkdir /licenses # buildkit       0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c chmod 0770 /usr/share/elastic…   0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    COPY /usr/share/elastic-agent /usr/share/ela…   1.73GB    buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c chmod 755 /usr/local/bin/dock…   426B      buildkit.dockerfile.v0
<missing>      8 weeks ago    COPY docker-entrypoint /usr/local/bin/docker…   426B      buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c set -e ;   TINI_BIN="";   TIN…   23.9kB    buildkit.dockerfile.v0
<missing>      8 weeks ago    ENV GODEBUG=madvdontneed=1                      0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    ENV PATH=/usr/share/elastic-agent:/usr/local…   0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    ENV ELASTIC_CONTAINER=true                      0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    LABEL org.label-schema.build-date=2023-09-07…   0B        buildkit.dockerfile.v0
<missing>      8 weeks ago    RUN /bin/sh -c for iter in {1..10}; do      …   49.6MB    buildkit.dockerfile.v0
<missing>      8 weeks ago    ENV BEAT_SETUID_AS=elastic-agent                0B        buildkit.dockerfile.v0
<missing>      3 months ago   /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B        
<missing>      3 months ago   /bin/sh -c #(nop) ADD file:ef6e767091d76c146…   65.7MB    
<missing>      3 months ago   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      3 months ago   /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B        
<missing>      3 months ago   /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B        
<missing>      3 months ago   /bin/sh -c #(nop)  ARG RELEASE                  0B        

The biggest difference in 8.11 is this line which was kBs in 8.10:

fe8243cd3cad      45 hours ago   RUN /bin/sh -c groupadd --gid 1000 elastic-a…   1.77GB    buildkit.dockerfile.v0
d500f2e33781      8 weeks ago    RUN /bin/sh -c groupadd --gid 1000 elastic-a…   1.68kB    buildkit.dockerfile.v0

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

This is because of #3576 which changed the affected lines.

[jlind23]

@pierrehilbert @cmacknz I believe this should be marked as a high severity issue and fixed asap.

[cmacknz]

Yes we should fix this, this is because of the additional chown command in #3576 which is creating a copy of the file system with the permissions change.

There is a good explanation of what is happening in https://medium.com/@mmornati/docker-images-and-files-chown-40d2f7248fcc

What about the chown? It is following the same rules: applying a change ownership to a file, for Docker means copy that file in the new layer and change the ownership. Anytime you are using it you are taking more disk space than you need.

[pchila]

We should probably rearrange the order of creation of the user, switch to such user and copy the files once

[AndersonQ]

also, perhaps we can copy and change the ownership at the same time. If the user is already created when we copy, it should be an easy fix (at least I think so, I haven't checked the docs though)

[cmacknz]

This has been reverted in 8.11 #3712 and will be reverted in main shortly

[cmacknz]

This problem is no longer present in 8.11.1, but we fixed it by reverting the fix for #82 which we need to bring back but without this bug. That is why this issue is still open.