[BUG] <extra_hosts array items[0,1] must be unique>

[GuillaumeCisco]

Description

Docker compose file is seen as invalid.

code is:

  foo:
    <<: *base
    container_name: foo
    build:
      context: ../foo
      <<: *docker-build-args
    command: [ 'nodemon', 'entrypoints/bugs' ]
    dns:
      - '8.8.8.8'
    dns_search: internal.namespace
    extra_hosts:
      - 'host.docker.internal:host-gateway'

Steps To Reproduce

$> docker compose version
Docker Compose version v2.29.6

error message:

validating /home/$USER/Projects/docker-compose.yml: services.foo.extra_hosts array items[0,1] must be unique

Compose Version

v2.29.6

Docker Environment

$> docker version
Client: Docker Engine - Community
 Version:           27.3.0
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        e85edf8
 Built:             Thu Sep 19 14:25:51 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Thu Sep 19 14:25:51 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Anything else?

Linked to #11371

[glours]

Hello @GuillaumeCisco
Can you share with us (after you removed the sensitive data) the output of docker compose config with Compose v2.29.6 and a previous working version
Currently without the values of the YAML anchor and without knowing if you have an override file present in your main directory, I'm not able to reproduce on my side

> docker compose version
Docker Compose version v2.29.6-desktop.1
> bat compose.yaml
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: compose.yaml
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ services:
   2   │   foo:
   3   │     container_name: foo
   4   │     build:
   5   │       context: ../yfoo
   6   │     command: [ 'nodemon', 'entrypoints/bugs' ]
   7   │     dns:
   8   │       - '8.8.8.8'
   9   │     dns_search: internal.namespace
  10   │     extra_hosts:
  11   │       - 'host.docker.internal:host-gateway'
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
> docker compose config
name: issue-12155
services:
  foo:
    build:
      context: /xxx/xxx/tmp/yfoo
      dockerfile: Dockerfile
    command:
      - nodemon
      - entrypoints/bugs
    container_name: foo
    dns:
      - 8.8.8.8
    dns_search:
      - internal.namespace
    extra_hosts:
      - host.docker.internal=host-gateway
    networks:
      default: null
networks:
  default:
    name: issue-12155_default

[GuillaumeCisco]

$> docker compose config
validating /home/g$USER/Projects/docker-compose.yml: services.foo.extra_hosts array items[0,1] must be unique

<<: *base refers to a file entitled docker-compose.base.yaml with inside:

services:
  base:
    extends:
      file: docker-compose.vars.yaml
      service: base-vars
    extra_hosts:
      - 'host.docker.internal:host-gateway'
    dns:
      - '8.8.8.8'
    dns_search: internal.namespace

If I comment this part:

    extra_hosts:
      - 'host.docker.internal:host-gateway'

It runs correctly

[ndeloof]

as base already declares extra_hosts why do you redeclare it in your foo service ?
this is the cause for this duplicate to be detected

note: in earlier release we only selected a single entry per host name, but this is wrong as extra_host can be repeated to provide alternate IP addresses, see #12088

[glours]

Can you show us how you declare your *base anchor and how it's linked to your docker-compose.base.yaml?
From what I see with your current definition, you'll try to inject

services:
  base:

in your foo service which can't be supported regarding the Compose Specification

Can you share with us the full list of resources involved in your configuration, so we'll be able to reproduce?

[GuillaumeCisco]

@ndeloof
Yes it is duplicated in both.
But what is interesting here is that only extra_hosts triggers the issue, dns and dns_search don't.
dns also can take an array.
A bit strange.
I fixed, by removing the duplication, but I don't think this issue should appear in the first place.

@glours no I cannot, and everything is always shared for reproduction.

[ndeloof]

extra_hosts is for sure an array but entries must be unique, or this doesn't make any sense.
Merge from multiple yaml structs used to drop such duplicates, but this prevented passing alternate IPs for same hostname

[g-ulli]

We are getting the same error. We do not have duplicated extra_hosts entries. Our setup:

docker-compose.yml

services:
    my-dev-container:
        image: ghcr.io/...
        depends_on:
            - my-mongodb
        # ...
        # Make host.docker.internal also work on linux
        extra_hosts:
            - "host.docker.internal:host-gateway"

    my-mongodb:
        image: mongo:latest
        expose:
            - 27017
        volumes:
            - db-data:/data/db
volumes:
    db-data:

# ...

docker-compose.devcontainer.yml

services:
    my-dev-container:
        extends:
            file: docker-compose.yml
            service: my-dev-container
        volumes:
            # Forwarding the SSH agent
            - ${SSH_AUTH_SOCK}:/ssh-agent
        environment:
            - SSH_AUTH_SOCK=/ssh-agent
        # security_opt block has to be defined here until this issue is fixed:
        # https://github.com/docker/compose/issues/11777
        security_opt:
            - seccomp:unconfined
            - apparmor:unconfined
volumes:
    # ...

Using the same workaround as for security_opt (moving it to the file that extends) works for extra_hosts, too.

This behavior changed in the last few days. Before, we did not have any issues with extra_hosts, only security_opt.

[glours]

Thanks @g-ulli I was able to reproduce with your example 👍