[BUG] Submount fails when defining configs in dependent service

[MikaelElkiaer]

Description

There seems to be some strange interconnections between volumes and configs, when starting a service as a dependency.

A service with a submount can start fine when running on its own - even when defining a config.
But starting a service that depends on the service with a submount will make the dependent service fail - if a config is defined for the dependent service.

Steps To Reproduce

Running this compose file:

configs:
  config:
    content: |
      config
services:
  volumer:
    command:
      - sh
      - -c
      - |
        mkdir -p /tmp/volume
        echo volume > /tmp/volume/test.txt
    image: alpine
    volumes:
      - tmp:/tmp
  without-config:
    command:
      - sh
      - -c
      - |
        cat /tmp/volume/test.txt
    depends_on:
      volumer:
        condition: service_completed_successfully
    image: alpine
    volumes:
      - type: volume
        source: tmp
        target: /tmp/volume
        volume:
          subpath: volume
        read_only: true
  without-config-too:
    command:
      - sh
      - -c
      - |
        echo touched-too
    depends_on:
      without-config:
        condition: service_completed_successfully
    image: alpine
  with-config:
    command:
      - sh
      - -c
      - |
        cat /tmp/volume/test.txt
        cat /tmp/config.txt
    # WARN: Seems to break the volume mount when run as a dependency
    configs:
      - source: config
        target: /tmp/config.txt
    depends_on:
      volumer:
        condition: service_completed_successfully
    image: alpine
    volumes:
      - type: volume
        source: tmp
        target: /tmp/volume
        volume:
          subpath: volume
        read_only: true
  with-config-too:
    command:
      - sh
      - -c
      - |
        echo touched-too
    depends_on:
      with-config:
        condition: service_completed_successfully
    image: alpine
volumes:
  tmp:

will fail if you run docker compose run --rm with-config-too, while it works when running any other service.
Remember to docker compose down --volumes when running different services.

Compose Version

Docker Compose version v2.29.2

Docker Environment

Client: Docker Engine - Community
 Version:    27.2.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 24
  Running: 17
  Paused: 0
  Stopped: 7
 Images: 98
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-23-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.13GiB
 Name: me-ks-workstation.alb.is.keysight.com
 ID: e6e0533f-adb9-4394-8be5-1b145ae14930
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[ndeloof]

The issue is that service_completed_successfully applies to starting container, not creating.
Compose creates both container with-config and volumer but will delay the former one after the latter completed.
As volume is fresh new the target subpath doesn't exists so engine fails to create container.
There's no way to manage this in Compose by this time

[MikaelElkiaer]

How does that explain the fact that the volume submount works without the config and only breaks for the service with?

[MikaelElkiaer]

Oh, and also the fact it is only a problem when starting a 3rd service which depends on the service with a config?

[ndeloof]

I can reproduce without relying on a 3rd service:

 machin docker compose up  with-config
[+] Running 3/0
 ✔ Network machin_default          Created                                                                                                                                                0.0s 
 ✔ Volume "machin_tmp"             Created                                                                                                                                                0.0s 
 ✔ Container machin-volumer-1      Created                                                                                                                                                0.0s 
 ⠋ Container machin-with-config-1  Creating                                                                                                                                               0.0s 
Error response from daemon: cannot access path /var/lib/docker/volumes/machin_tmp/_data/volume: lstat /var/lib/docker/volumes/machin_tmp/_data/volume: no such file or directory

How does that explain the fact that the volume submount works without the config and only breaks for the service with?

I guess the root cause is collision between /tmp/config.txt and /tmp/volume, as engine tries to define mount options.
This is not a Compose issue: can be reproduced with plain docker run command:

$ docker run --mount source=tmp,target=/tmp/volume,volume-subpath=volume  alpine  ls -al /tmp
docker: Error response from daemon: cannot access path /var/lib/docker/volumes/tmp/_data/volume: lstat /var/lib/docker/volumes/tmp/_data/volume: no such file or directory.

[ndeloof]

I guess the reason you only detect this with a config set is that, as Compose relies on docker cp to create this config, the engine need to create the mounts for the container (which it uses to do only on start). When you dont, and have a dependency on volumer then the target subpath is created by volumer container then service starts and mounts are ok

[MikaelElkiaer]

Any suggestions on how to move forward?
Is this behaviour that could be changed?

[ndeloof]

Any suggestions on how to move forward?

Would need to better know your actual use-case to offer guidance.

Is this behaviour that could be changed?

This may have impacts so I can't tell. As a workaround, you can consider using a plain file as config, which would result into a bind mount

[ndeloof]

An actual fix would require to change up implementation so that it creates and start containers before it creates dependent ones. This would have impact on the UX as then we can't display the progress UX as such an initial container would dump logs to the console, still this is something to consider