Containers using publish all occasionally use the wrong port mapping for IPv6 binding

[tomelliff]

I've spent a chunk of time trying to chase down an issue we're seeing on some of our AWS ECS hosts where healthchecks were spuriously failing and causing containers to exit. I noticed from looking at the task state change events that somehow we had containers binding to different ports between IPv4 and IPv6 and all of our issues were caused by a mismatch here.

Rolling back to Docker 20.10.5 fixed this by not publishing IPv6 bindings for our containers at all (essentially undoing the work done in https://github.com/moby/libnetwork/pull/2608/files) but while we probably can avoid using IPv6 bindings either in Docker or in AWS' ECS Agent (According to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html#task-networking-vpc-dual-stack IPv6 shouldn't be happening unless opted into anyway so I think this is a surprise caused by Docker 20.10.6 now showing IPv6 bindings in the API when it didn't previously) but that doesn't escape the fact that there's a bug somewhere in libnetwork (I think, I'm a bit hazy on where responsibilities lie now with the splitting of the projects) that is causing Docker to publish the wrong ports on IPv6.

Running the following command:

xargs -P 0 -I {} docker run --detach --rm --name test-{} -P nginx<<<$(seq 1 100)

attempts to launch 100 NGINX containers in parallel which makes my laptop completely fall over and most of the container starts to fail. However when I run this I reliably get port mismatches. Sorting the docker ps output by the IPv4 address shows this well:

CONTAINER ID	IMAGE	COMMAND	PORTS		NAMES
bb9c4984b395	nginx	/docker-entrypoint.…	0.0.0.0:49164->80/tcp,	:::49164->80/tcp	test-2
571041910b22	nginx	/docker-entrypoint.…	0.0.0.0:49165->80/tcp,	:::49165->80/tcp	test-6
6d6379bf998f	nginx	/docker-entrypoint.…	0.0.0.0:49166->80/tcp,	:::49166->80/tcp	test-4
c6fa6a39c31d	nginx	/docker-entrypoint.…	0.0.0.0:49167->80/tcp,	:::49167->80/tcp	test-15
82e49c9bbb0f	nginx	/docker-entrypoint.…	0.0.0.0:49168->80/tcp,	:::49168->80/tcp	test-11
b903859bdb9c	nginx	/docker-entrypoint.…	0.0.0.0:49169->80/tcp,	:::49169->80/tcp	test-8
c568a230b5b9	nginx	/docker-entrypoint.…	0.0.0.0:49170->80/tcp,	:::49170->80/tcp	test-12
78d13f523722	nginx	/docker-entrypoint.…	0.0.0.0:49171->80/tcp,	:::49171->80/tcp	test-3
aeef1fa41148	nginx	/docker-entrypoint.…	0.0.0.0:49172->80/tcp,	:::49172->80/tcp	test-10
2b9cfea3221c	nginx	/docker-entrypoint.…	0.0.0.0:49173->80/tcp,	:::49173->80/tcp	test-16
1174325c7394	nginx	/docker-entrypoint.…	0.0.0.0:49174->80/tcp,	:::49174->80/tcp	test-5
7258d99985b6	nginx	/docker-entrypoint.…	0.0.0.0:49175->80/tcp,	:::49175->80/tcp	test-14
470391662adb	nginx	/docker-entrypoint.…	0.0.0.0:49176->80/tcp,	:::49176->80/tcp	test-9
3aa55932f786	nginx	/docker-entrypoint.…	0.0.0.0:49177->80/tcp,	:::49177->80/tcp	test-17
fdad706caea0	nginx	/docker-entrypoint.…	0.0.0.0:49178->80/tcp,	:::49178->80/tcp	test-13
ac893be13721	nginx	/docker-entrypoint.…	0.0.0.0:49179->80/tcp,	:::49179->80/tcp	test-1
d16c3bc3db80	nginx	/docker-entrypoint.…	0.0.0.0:49180->80/tcp,	:::49180->80/tcp	test-20
c159beb98784	nginx	/docker-entrypoint.…	0.0.0.0:49181->80/tcp,	:::49181->80/tcp	test-19
0d449991367b	nginx	/docker-entrypoint.…	0.0.0.0:49182->80/tcp,	:::49182->80/tcp	test-7
6a99f572659a	nginx	/docker-entrypoint.…	0.0.0.0:49183->80/tcp,	:::49183->80/tcp	test-18
6cdc6c1aee92	nginx	/docker-entrypoint.…	0.0.0.0:49184->80/tcp,	:::49184->80/tcp	test-21
9ac258de0f00	nginx	/docker-entrypoint.…	0.0.0.0:49185->80/tcp,	:::49185->80/tcp	test-22
a753a81defcc	nginx	/docker-entrypoint.…	0.0.0.0:49186->80/tcp,	:::49186->80/tcp	test-25
2f8543fbc1ae	nginx	/docker-entrypoint.…	0.0.0.0:49187->80/tcp,	:::49187->80/tcp	test-27
bb1a37f274e0	nginx	/docker-entrypoint.…	0.0.0.0:49188->80/tcp,	:::49188->80/tcp	test-30
787d4286597d	nginx	/docker-entrypoint.…	0.0.0.0:49189->80/tcp,	:::49189->80/tcp	test-24
cfca7dbe2815	nginx	/docker-entrypoint.…	0.0.0.0:49191->80/tcp,	:::49191->80/tcp	test-31
93e001248f3a	nginx	/docker-entrypoint.…	0.0.0.0:49192->80/tcp,	:::49192->80/tcp	test-28
723055c393d6	nginx	/docker-entrypoint.…	0.0.0.0:49193->80/tcp,	:::49193->80/tcp	test-29
336a0e93ca3b	nginx	/docker-entrypoint.…	0.0.0.0:49194->80/tcp,	:::49194->80/tcp	test-26
6f404ff119b9	nginx	/docker-entrypoint.…	0.0.0.0:49195->80/tcp,	:::49195->80/tcp	test-33
61efecef0103	nginx	/docker-entrypoint.…	0.0.0.0:49196->80/tcp,	:::49196->80/tcp	test-35
1a1633e3563c	nginx	/docker-entrypoint.…	0.0.0.0:49197->80/tcp,	:::49197->80/tcp	test-32
f00b4f4b3e43	nginx	/docker-entrypoint.…	0.0.0.0:49198->80/tcp,	:::49198->80/tcp	test-39
b703b502cb0d	nginx	/docker-entrypoint.…	0.0.0.0:49199->80/tcp,	:::49199->80/tcp	test-65
95cd8136cb78	nginx	/docker-entrypoint.…	0.0.0.0:49200->80/tcp,	:::49200->80/tcp	test-66
025aa7cd9e81	nginx	/docker-entrypoint.…	0.0.0.0:49201->80/tcp,	:::49201->80/tcp	test-37
8067402d8030	nginx	/docker-entrypoint.…	0.0.0.0:49202->80/tcp,	:::49202->80/tcp	test-41
1.79769313486232E+308	nginx	/docker-entrypoint.…	0.0.0.0:49203->80/tcp,	:::49203->80/tcp	test-40
8dc601214115	nginx	/docker-entrypoint.…	0.0.0.0:49204->80/tcp,	:::49204->80/tcp	test-42
b32b03f7d1e8	nginx	/docker-entrypoint.…	0.0.0.0:49205->80/tcp,	:::49205->80/tcp	test-49
e5733efe3ac5	nginx	/docker-entrypoint.…	0.0.0.0:49206->80/tcp,	:::49206->80/tcp	test-44
f433ea6d4085	nginx	/docker-entrypoint.…	0.0.0.0:49208->80/tcp,	:::49207->80/tcp	test-46
364e5ecbd186	nginx	/docker-entrypoint.…	0.0.0.0:49209->80/tcp,	:::49208->80/tcp	test-43
2276f9cada2e	nginx	/docker-entrypoint.…	0.0.0.0:49210->80/tcp,	:::49209->80/tcp	test-36
9ee5b8acc20b	nginx	/docker-entrypoint.…	0.0.0.0:49211->80/tcp,	:::49210->80/tcp	test-38
1cec14c1e094	nginx	/docker-entrypoint.…	0.0.0.0:49212->80/tcp,	:::49211->80/tcp	test-50
aa3d7b553cd3	nginx	/docker-entrypoint.…	0.0.0.0:49215->80/tcp,	:::49214->80/tcp	test-55
4593e5efdc49	nginx	/docker-entrypoint.…	0.0.0.0:49216->80/tcp,	:::49215->80/tcp	test-56
5f37baa5f433	nginx	/docker-entrypoint.…	0.0.0.0:49217->80/tcp,	:::49216->80/tcp	test-67
5ade3066ffef	nginx	/docker-entrypoint.…	0.0.0.0:49218->80/tcp,	:::49217->80/tcp	test-53
4610422e9af1	nginx	/docker-entrypoint.…	0.0.0.0:49220->80/tcp,	:::49219->80/tcp	test-99
0a2f89796ce0	nginx	/docker-entrypoint.…	0.0.0.0:49222->80/tcp,	:::49221->80/tcp	test-51

Container f433ea6d4085 is mapped to host port 49208 on IPv4 but mapped to host port 49207 for IPv6. This isn't an issue by itself as the host is essentially listening on two ports for the same container port but if you then look at the next container 364e5ecbd186 which is mapped to host port 49209 for IPv4 and 49208 for IPv6. So requests to this container on IPv6 will instead hit the other container. In this case with the containers all running the same image that's fine but where you have different containers then the request will fail as it will be mapped to the wrong container. The same repeats down the list so that all the next containers are then off by one throughout.