Docker DNS Service Discovery "no such host"

[SIMULATAN]

I'm running two mimir nodes in a stack and want to use DNS Service Discovery to connect the two nodes together.
Therefore, I configured dns+tasks.monitoring_mimir as a join_member in mimir.yml.
Unfortunately, when it tries to do the DNS query, this is printed in the console:

caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=tasks.monitoring_mimir err="lookup tasks.monitoring_mimir on 127.0.0.11:53: no such host"
caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=tasks.monitoring_mimir

However, when running dig tasks.monitoring_mimir, it works just as expected!

(both when doing a docker exec and using that as the entrypoint)

Also, when looking at the dashboard, I can see that the member for some reason uses the wrong interface?
That one is the bridge interface made when publishing ports.

I tried setting the name manually to the correct interface name (eth0) by using instance_interface_names, but it's no use.

Logfile
mimir.yml
docker-compose.yml

I tried multiple combinations of using instance_interface_names, instance_addr, service discovery prefixes, network drivers, nodes, architectures, replication modes and more, but nothing worked, it keeps erroring with the above logs.

Another observation was that it tries to connect to some IPs in the 192.168.x.x subnet, even though I didn't add that range anywhere and the DNS query doesn't return those IPs.

[pracucci]

caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=tasks.monitoring_mimir err="lookup tasks.monitoring_mimir on 127.0.0.11:53: no such host"
caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=tasks.monitoring_mimir

I'm not sure what's happening here. The configuration you shared looks correct to me. Since I'm not familiar with your setup, let me ask a dumb question: can you confirm the host tasks.monitoring_mimir exists before Mimir process starts, right?

Also, when looking at the dashboard, I can see that the member for some reason uses the wrong interface?
That one is the bridge interface made when publishing ports.

The IP address exposed by memberlist can be configured setting -memberlist.advertise-addr or its respective YAML config option (doc). The default is using the address from the same interface where the memberlist listening port was binded, which is configured by -memberlist.bind-addr.

[SIMULATAN]

Thanks for the response!

can you confirm the host tasks.monitoring_mimir exists before Mimir process starts, right?

Well, depends. I did try setting the dig command as the entrypoint (=> it runs pretty much instantly, or, at least faster than mimir's DNS lookup) and it worked as expected.

About the advertise addr; can I also tell it to bind to a specific interface? Since I'm using docker and dynamically created networks I try to not hard code any IP (ranges) but rather let docker decide what IPs to give out. After restarting a few times, I did observe that the interface name stays consistent (even with multiple networks attached to the container). I saw that instance_interface_names exists for some objects but not for KVConfig apparently.
Also, I even if I hardcoded the range, I couldn't use these two settings as the cluster scales with docker swarm and thus the config is the same for all instances.

Do I even need advertising or can that just be bypassed by using DNS service discovery?

Oh, also, about the DNS lookup; as you can see in the screenshot, tasks.monitoring_mimir doesn't resolve to a specific single host but rather to multiple ones (including the current instance), I checked the docs on that but since dig seems to return A results and the query was an A one too dns+ should be the right service discovery method, right..?

I can also observe that the second instance does pop up in the dashboard for around 30 seconds, after which it just disappears and is gone forever with the KV Store Values rising linearly.

Anyway, I went ahead and put my logs into a gist to avoid overlooking important details.
https://gist.github.com/SIMULATAN/c529220bc236e6f8e04332777d215c3e

Thanks a ton for the help!

[SIMULATAN]

Well, I managed to fix the IP issue by introducing a hack that allows this whole thing to be dynamically scalable but still work just fine.

I just used this as an entrypoint:
/entrypoint.sh

#!/bin/sh
IP="$(ip addr show eth1 | grep inet | tr -s ' ' | cut -f 3 -d ' ' | cut -f 1 -d '/')"
cat /etc/mimir/config.yml | sed 's/%IP%/'"$IP"'/' | tee /etc/mimir/config-final.yml
/bin/mimir --config.file=/etc/mimir/config-final.yml --config.expand-env

This checks the interface of the overlay network, gets the IP from it and substitutes all cases of %IP% in my mimir.yml
This works because I set all the instance_addrs available to %IP% to be substituted with the actual IP when the container boots up.

Now, the instances connect together just fine!

When looking at the ring status of the different components, it shows both instances with a roughly 50 - 50 ratio of ownership, so I'd assume it works as expected..?

mimir.log (includes the mimir.yml with substituted values at the top)
For some reason, the DNS query still fails, but it finds the other instances anyway...

[budiprihhastomo]

The same issue that I am experiencing, is there any fix or improvement regarding Mimir in the Docker Swarm environment?

[marvinnitz18]

i seem to have the exact same problem, but eth1 would be wrong for me, eth0 would be correct:

i will try to implement the workaround @SIMULATAN has provided now

[marvinnitz18]

for everyone using docker swarm, this i my solution i put the workaround into a wrapper.sh which i mounted into the container and defined it as the entrypoint in the stackfile:
version: "3.7" services: mimir1: image: grafana/mimir:2.11.0 entrypoint: ["/bin/wrapper.sh"]

wrapper.sh:
#!/bin/sh IP="$(ip addr show eth0 | grep inet | tr -s ' ' | cut -f 3 -d ' ' | cut -f 1 -d '/')" exec /bin/mimir -config.file=/etc/mimir.yaml -memberlist.advertise-addr=$IP

[ksingh-scogo]

@SIMULATAN @pracucci encountering exact same issue with Mimir running on K8s

ts=2023-12-24T12:18:35.380649131Z caller=log.go:245 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
ts=2023-12-24T12:18:35.381232016Z caller=tcp_transport.go:405 level=debug component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=10.244.2.176 advertisePort=7946
ts=2023-12-24T12:18:35.381724232Z caller=tcp_transport.go:405 level=debug component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=10.244.2.176 advertisePort=7946

ts=2023-12-24T12:18:35.38402796Z caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=mimir-gossip-ring.grafana.svc.cluster.local. err="lookup mimir-gossip-ring.grafana.svc.cluster.local. on 10.0.0.10:53: no such host"

ts=2023-12-24T12:18:35.384052997Z caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=mimir-gossip-ring.grafana.svc.cluster.local.

ts=2023-12-24T12:18:35.384066923Z caller=memberlist_client.go:541 level=info msg="memberlist fast-join starting" nodes_found=0 to_join=0
ts=2023-12-24T12:18:35.384078274Z caller=memberlist_client.go:561 level=warn msg="memberlist fast-join finished" joined_nodes=0 elapsed_time=2.201117ms

ts=2023-12-24T12:18:35.384090126Z caller=memberlist_client.go:573 level=info phase=startup msg="joining memberlist cluster" join_members=dns+mimir-gossip-ring.grafana.svc.cluster.local.:7946

ts=2023-12-24T12:18:35.385085489Z caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=mimir-gossip-ring.grafana.svc.cluster.local. err="lookup mimir-gossip-ring.grafana.svc.cluster.local. on 10.0.0.10:53: no such host"

ts=2023-12-24T12:18:35.385100887Z caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=mimir-gossip-ring.grafana.svc.cluster.local.

ts=2023-12-24T12:18:35.3851131Z caller=memberlist_client.go:607 level=warn phase=startup msg="joining memberlist cluster" attempts=1 max_attempts=10 err="found no nodes to join"

[ksingh-scogo]

i am using mimir helm

[pracucci]

From the logs we can see:

err="lookup mimir-gossip-ring.grafana.svc.cluster.local. on 10.0.0.10:53: no such host"

I suggest to run few sanity checks to begin with:

1. Does the mimir-gossip-ring K8S service exists in the grafana namespace? It should exist, as it's expected to be created as part of the Helm chart.
2. If you get a shell in one of the Mimir pods, can they resolve the service host mimir-gossip-ring.grafana.svc.cluster.local?