Unable to connect to Milvus deployed on a GKE cluster #36002

matsuo1234567 · 2024-09-05T07:37:44Z

matsuo1234567
Sep 5, 2024

I am stuck here.
https://github.com/milvus-io/milvus-docs/blob/v2.4.x/site/en/adminGuide/clouds/gcp/gcp.md#hello-milvus

Error

% python hello_milvus.py                                                        

=== start connecting to Milvus     ===

Traceback (most recent call last):
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 147, in _wait_for_channel_ready
    grpc.channel_ready_future(self._channel).result(timeout=timeout)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/grpc/_utilities.py", line 162, in result
    self._block(timeout)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/grpc/_utilities.py", line 106, in _block
    raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/matsuo/dev/milvus-test/hello_milvus.py", line 32, in <module>
    connections.connect("default", host="35.243.101.14", port="19530")
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/orm/connections.py", line 447, in connect
    connect_milvus(**kwargs, user=user, password=password, token=token, db_name=db_name)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/orm/connections.py", line 398, in connect_milvus
    gh._wait_for_channel_ready(timeout=timeout)
  File "/Users/matsuo/.pyenv/versions/3.10.4/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 150, in _wait_for_channel_ready
    raise MilvusException(
pymilvus.exceptions.MilvusException: <MilvusException: (code=2, message=Fail connecting to server on **.***.***.**:19530, illegal connection params or server unavailable)>

% kubectl get services|grep my-release-milvus|grep LoadBalancer|awk '{print $4}'

**.***.***.**

hello_milvus.py
connections.connect("default", host="**.***.***.**", port="19530")

% kubectl get services
                                                          
NAME                          TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                               AGE
kubernetes                    ClusterIP      10.104.64.1     <none>          443/TCP                               6h43m
my-release-etcd               ClusterIP      10.104.67.205   <none>          2379/TCP,2380/TCP                     37m
my-release-etcd-headless      ClusterIP      None            <none>          2379/TCP,2380/TCP                     37m
my-release-milvus             LoadBalancer   10.104.67.174   **.***.***.**   19530:30860/TCP,9091:32046/TCP        20m
my-release-milvus-datanode    ClusterIP      None            <none>          9091/TCP                              37m
my-release-milvus-indexnode   ClusterIP      None            <none>          9091/TCP                              37m
my-release-milvus-mixcoord    ClusterIP      10.104.67.93    <none>          9091/TCP                              37m
my-release-milvus-querynode   ClusterIP      None            <none>          9091/TCP                              37m
my-release-pulsar-bookie      ClusterIP      None            <none>          3181/TCP,8000/TCP                     37m
my-release-pulsar-broker      ClusterIP      None            <none>          8080/TCP,6650/TCP                     37m
my-release-pulsar-proxy       ClusterIP      10.104.64.142   <none>          8080/TCP,6650/TCP                     37m
my-release-pulsar-recovery    ClusterIP      None            <none>          8000/TCP                              37m
my-release-pulsar-zookeeper   ClusterIP      None            <none>          8000/TCP,2888/TCP,3888/TCP,2181/TCP   37m

values.yaml

cluster:
    enabled: true

service:
    type: LoadBalancer

minio:
    enabled: false
    tls:
        enabled: false

externalS3:
    enabled: true
    host: storage.googleapis.com
    port: 443
    rootPath: milvus/my-release
    bucketName: milvus-test-bucket
    cloudProvider: gcp
    useSSL: true
    accessKey: "**"
    secretKey: "**"

yhmo · 2024-09-05T08:46:48Z

yhmo
Sep 5, 2024
Collaborator

Check the log of the mixcoord.

3 replies

matsuo1234567 Sep 5, 2024
Author

my-release-milvus-mixcoord-867bb7d6fd-mmgmt.log
my-release-milvus-mixcoord-867bb7d6fd-mmgmt.log
my-release-milvus-mixcoord-867bb7d6fd-mmgmt.log

yhmo Sep 5, 2024
Collaborator

Seems no problem for the mixcoord.
Is the port 19530 redirected to 30860 ? "19530:30860/TCP"
Check the log of the proxy(my-release-milvus) pod.

matsuo1234567 Sep 5, 2024
Author

my-release-milvus-proxy-65596c894d-fdtzv.log
my-release-milvus-proxy-65596c894d-fdtzv.log
my-release-milvus-proxy-65596c894d-fdtzv.log

xiaofan-luan · 2024-09-05T23:47:52Z

xiaofan-luan
Sep 5, 2024
Maintainer

[2024/09/05 06:54:26.402 +00:00] [INFO] [proxy/service.go:503] ["init Proxy's tracer done"] ["service name"="Proxy ip: 10.104.77.155, port: 19530"]
[2024/09/05 06:54:26.403 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-release-etcd-0.my-release-etcd-headless.default.svc.cluster.local:2379,my-release-etcd-1.my-release-etcd-headless.default.svc.cluster.local:2379,my-release-etcd-2.my-release-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/09/05 06:54:26.412 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #5 SubChannel #7] grpc: addrConn.createTransport failed to connect to {Addr: "my-release-etcd-1.my-release-etcd-headless.default.svc.cluster.local:2379", ServerName: "my-release-etcd-1.my-release-etcd-headless.default.svc.cluster.local", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.104.73.24:2379: connect: connection refused""]

5 replies

xiaofan-luan Sep 5, 2024
Maintainer

seem that the proxy can not connect to 10.104.73.24:2379, which is your etcd server

matsuo1234567 Sep 5, 2024
Author

Thank you for checking logs.

Is this?

% kubectl describe pod my-release-etcd-1  
Name:             my-release-etcd-1
Namespace:        default
Priority:         0
Service Account:  default
Node:             gke-milvus-distribut-milvus-distribut-89938ced-z7po/172.22.165.203
Start Time:       Thu, 05 Sep 2024 15:54:08 +0900
Labels:           app.kubernetes.io/instance=my-release
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=etcd
                  apps.kubernetes.io/pod-index=1
                  controller-revision-hash=my-release-etcd-59465bbc7b
                  helm.sh/chart=etcd-6.3.3
                  statefulset.kubernetes.io/pod-name=my-release-etcd-1
Annotations:      <none>
Status:           Running
IP:               10.104.73.24
IPs:
  IP:           10.104.73.24
Controlled By:  StatefulSet/my-release-etcd
Containers:
  etcd:
    Container ID:   containerd://54a1adb77085613eeebc9613957944febb2aba217922e7000dbf7ef31584124b
    Image:          docker.io/milvusdb/etcd:3.5.14-r1
    Image ID:       docker.io/milvusdb/etcd@sha256:7fc0fd9ad6d2ea14e245b7e7c7565e2630b76af2682ba320647239d21fd4fbcb
    Ports:          2379/TCP, 2380/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 05 Sep 2024 15:54:25 +0900
    Ready:          True
    Restart Count:  0
    Liveness:       exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=10s period=30s #success=1 #failure=5
    Readiness:      exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=10s period=20s #success=1 #failure=5
    Environment:
      BITNAMI_DEBUG:                     false
      MY_POD_IP:                          (v1:status.podIP)
      MY_POD_NAME:                       my-release-etcd-1 (v1:metadata.name)
      ETCDCTL_API:                       3
      ETCD_ON_K8S:                       yes
      ETCD_START_FROM_SNAPSHOT:          no
      ETCD_DISASTER_RECOVERY:            no
      ETCD_NAME:                         $(MY_POD_NAME)
      ETCD_DATA_DIR:                     /bitnami/etcd/data
      ETCD_LOG_LEVEL:                    info
      ALLOW_NONE_AUTHENTICATION:         yes
      ETCD_ADVERTISE_CLIENT_URLS:        http://$(MY_POD_NAME).my-release-etcd-headless.default.svc.cluster.local:2379
      ETCD_LISTEN_CLIENT_URLS:           http://0.0.0.0:2379
      ETCD_INITIAL_ADVERTISE_PEER_URLS:  http://$(MY_POD_NAME).my-release-etcd-headless.default.svc.cluster.local:2380
      ETCD_LISTEN_PEER_URLS:             http://0.0.0.0:2380
      ETCD_AUTO_COMPACTION_MODE:         revision
      ETCD_AUTO_COMPACTION_RETENTION:    1000
      ETCD_INITIAL_CLUSTER_TOKEN:        etcd-cluster-k8s
      ETCD_INITIAL_CLUSTER_STATE:        new
      ETCD_INITIAL_CLUSTER:              my-release-etcd-0=http://my-release-etcd-0.my-release-etcd-headless.default.svc.cluster.local:2380,my-release-etcd-1=http://my-release-etcd-1.my-release-etcd-headless.default.svc.cluster.local:2380,my-release-etcd-2=http://my-release-etcd-2.my-release-etcd-headless.default.svc.cluster.local:2380
      ETCD_CLUSTER_DOMAIN:               my-release-etcd-headless.default.svc.cluster.local
      ETCD_QUOTA_BACKEND_BYTES:          4294967296
      ETCD_HEARTBEAT_INTERVAL:           500
      ETCD_ELECTION_TIMEOUT:             2500
    Mounts:
      /bitnami/etcd from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vg5vm (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-my-release-etcd-1
    ReadOnly:   false
  kube-api-access-vg5vm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

xiaofan-luan Sep 6, 2024
Maintainer

you can try to use a etcd cli to check if you etcd service is on.
right not I saw 10.104.73.24 can not be connected. Not sure if this is because of etcd can not brought up or there is a network issue here.

matsuo1234567 Sep 6, 2024
Author

It seems etcd service is off.

% export ETCDCTL_API=3   
% export ETCDCTL_ENDPOINTS=http://10.104.73.24:2379
% etcdctl endpoint status
{"level":"warn","ts":"2024-09-06T09:57:28.953121+0900","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000626000/10.104.73.24:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint http://10.104.73.24:2379 (context deadline exceeded)
% etcdctl endpoint health 
{"level":"warn","ts":"2024-09-06T09:57:39.861463+0900","logger":"client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0003b2000/10.104.73.24:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
http://10.104.73.24:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster

matsuo1234567 Sep 6, 2024
Author

Why when I run % kubectl describe pod my-release-etcd-1, staratus is running but the etcd service is not running?

matsuo1234567 · 2024-09-06T08:29:39Z

matsuo1234567
Sep 6, 2024
Author

Below is the log of my-release-milvus-etcd-1 warnings.
Is this relevant?

{"level":"warn","ts":"2024-09-05T06:54:40.722912Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2024-09-05T06:54:40.72294Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2024-09-05T06:54:40.722956Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2024-09-05T06:54:40.722962Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2024-09-05T06:54:40.722975Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2024-09-05T06:54:40.722983Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2024-09-05T06:54:40.723008Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=my-release-etcd-headless.default.svc.cluster.local"}
{"level":"warn","ts":"2024-09-05T06:54:40.723021Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2024-09-05T06:54:40.723033Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2024-09-05T06:54:40.723049Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2024-09-05T06:54:40.723059Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2024-09-05T06:54:40.723071Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2024-09-05T06:54:40.72308Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2024-09-05T06:54:40.723094Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2024-09-05T06:54:40.72311Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2024-09-05T06:54:40.723124Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2024-09-05T06:54:40.723139Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2024-09-05T06:54:40.72315Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2024-09-05T06:54:40.72316Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"warn","ts":"2024-09-05T06:54:40.723457Z","caller":"embed/config.go:687","msg":"Running http and grpc server on single port. This is not recommended for production."}

{"level":"warn","ts":"2024-09-05T06:54:40.723733Z","caller":"embed/config.go:687","msg":"Running http and grpc server on single port. This is not recommended for production."}

{"level":"warn","ts":"2024-09-05T06:54:41.827483Z","caller":"auth/store.go:1241","msg":"simple token is not cryptographically signed"}

What is the problem?

% kubectl exec -it my-release-etcd-1 -- env     

PATH=/opt/bitnami/common/bin:/opt/bitnami/etcd/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=my-release-etcd-1
HOME=/
OS_ARCH_RAW=linux/amd64
OS_FLAVOUR=debian-11
OS_NAME=linux
APP_VERSION=3.5.14
BITNAMI_APP_NAME=etcd
ETCDCTL_API=3
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_CLUSTER_DOMAIN=my-release-etcd-headless.default.svc.cluster.local
ETCD_HEARTBEAT_INTERVAL=500
MY_POD_NAME=my-release-etcd-1
ETCD_DISASTER_RECOVERY=no
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://my-release-etcd-1.my-release-etcd-headless.default.svc.cluster.local:2380
ETCD_AUTO_COMPACTION_RETENTION=1000
ETCD_LOG_LEVEL=info
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_AUTO_COMPACTION_MODE=revision
ETCD_INITIAL_CLUSTER=my-release-etcd-0=http://my-release-etcd-0.my-release-etcd-headless.default.svc.cluster.local:2380,my-release-etcd-1=http://my-release-etcd-1.my-release-etcd-headless.default.svc.cluster.local:2380,my-release-etcd-2=http://my-release-etcd-2.my-release-etcd-headless.default.svc.cluster.local:2380
ETCD_ELECTION_TIMEOUT=2500
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
ETCD_QUOTA_BACKEND_BYTES=4294967296
BITNAMI_DEBUG=false
MY_POD_IP=10.104.73.24
ETCD_ON_K8S=yes
ETCD_NAME=my-release-etcd-1
ETCD_ADVERTISE_CLIENT_URLS=http://my-release-etcd-1.my-release-etcd-headless.default.svc.cluster.local:2379
ETCD_START_FROM_SNAPSHOT=no
ETCD_DATA_DIR=/bitnami/etcd/data
ALLOW_NONE_AUTHENTICATION=yes
MY_RELEASE_ETCD_SERVICE_HOST=10.104.67.205
MY_RELEASE_ETCD_PORT_2380_TCP_PORT=2380
KUBERNETES_PORT_443_TCP_PORT=443
MY_RELEASE_MILVUS_MIXCOORD_PORT_9091_TCP_PORT=9091
MY_RELEASE_PULSAR_PROXY_PORT_8080_TCP=tcp://10.104.64.142:8080
MY_RELEASE_PULSAR_PROXY_PORT_8080_TCP_PROTO=tcp
MY_RELEASE_ETCD_SERVICE_PORT_CLIENT=2379
KUBERNETES_SERVICE_HOST=10.104.64.1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=10.104.64.1
MY_RELEASE_MILVUS_MIXCOORD_SERVICE_HOST=10.104.67.93
KUBERNETES_PORT_443_TCP=tcp://10.104.64.1:443
MY_RELEASE_ETCD_PORT_2379_TCP=tcp://10.104.67.205:2379
KUBERNETES_PORT=tcp://10.104.64.1:443
MY_RELEASE_MILVUS_SERVICE_HOST=10.104.70.106
MY_RELEASE_MILVUS_PORT_9091_TCP_PROTO=tcp
MY_RELEASE_PULSAR_PROXY_PORT_8080_TCP_PORT=8080
MY_RELEASE_ETCD_PORT_2379_TCP_PORT=2379
MY_RELEASE_PULSAR_PROXY_SERVICE_PORT_HTTP=8080
MY_RELEASE_PULSAR_PROXY_SERVICE_PORT_PULSAR=6650
MY_RELEASE_PULSAR_PROXY_PORT=tcp://10.104.64.142:8080
MY_RELEASE_MILVUS_MIXCOORD_PORT_9091_TCP_PROTO=tcp
MY_RELEASE_ETCD_SERVICE_PORT=2379
MY_RELEASE_ETCD_PORT_2380_TCP_PROTO=tcp
MY_RELEASE_ETCD_PORT_2380_TCP_ADDR=10.104.67.205
MY_RELEASE_MILVUS_MIXCOORD_PORT_9091_TCP_ADDR=10.104.67.93
MY_RELEASE_PULSAR_PROXY_SERVICE_PORT=8080
MY_RELEASE_PULSAR_PROXY_PORT_6650_TCP_ADDR=10.104.64.142
MY_RELEASE_ETCD_PORT_2380_TCP=tcp://10.104.67.205:2380
MY_RELEASE_PULSAR_PROXY_SERVICE_HOST=10.104.64.142
MY_RELEASE_PULSAR_PROXY_PORT_6650_TCP_PORT=6650
MY_RELEASE_ETCD_SERVICE_PORT_PEER=2380
MY_RELEASE_ETCD_PORT_2379_TCP_PROTO=tcp
MY_RELEASE_MILVUS_PORT_19530_TCP_PROTO=tcp
MY_RELEASE_MILVUS_PORT_9091_TCP_PORT=9091
MY_RELEASE_MILVUS_PORT_9091_TCP=tcp://10.104.70.106:9091
MY_RELEASE_ETCD_PORT=tcp://10.104.67.205:2379
MY_RELEASE_ETCD_PORT_2379_TCP_ADDR=10.104.67.205
KUBERNETES_SERVICE_PORT=443
MY_RELEASE_MILVUS_SERVICE_PORT=19530
MY_RELEASE_MILVUS_SERVICE_PORT_MILVUS=19530
MY_RELEASE_MILVUS_SERVICE_PORT_METRICS=9091
MY_RELEASE_MILVUS_PORT_19530_TCP_ADDR=10.104.70.106
MY_RELEASE_MILVUS_MIXCOORD_PORT=tcp://10.104.67.93:9091
MY_RELEASE_MILVUS_MIXCOORD_PORT_9091_TCP=tcp://10.104.67.93:9091
MY_RELEASE_MILVUS_PORT=tcp://10.104.70.106:19530
MY_RELEASE_MILVUS_PORT_19530_TCP_PORT=19530
MY_RELEASE_MILVUS_PORT_9091_TCP_ADDR=10.104.70.106
KUBERNETES_SERVICE_PORT_HTTPS=443
MY_RELEASE_MILVUS_MIXCOORD_SERVICE_PORT_METRICS=9091
MY_RELEASE_PULSAR_PROXY_PORT_8080_TCP_ADDR=10.104.64.142
MY_RELEASE_PULSAR_PROXY_PORT_6650_TCP=tcp://10.104.64.142:6650
MY_RELEASE_MILVUS_MIXCOORD_SERVICE_PORT=9091
MY_RELEASE_MILVUS_PORT_19530_TCP=tcp://10.104.70.106:19530
MY_RELEASE_PULSAR_PROXY_PORT_6650_TCP_PROTO=tcp
TERM=xterm

4 replies

LoveEachDay Sep 6, 2024
Maintainer

@matsuo1234567 Could you bring the full etcd pod log?

matsuo1234567 Sep 9, 2024
Author

my-release-etcd-0.log
my-release-etcd-1.log
my-release-etcd-2.log

LoveEachDay Sep 9, 2024
Maintainer

@matsuo1234567 Based on the logs of all etcd pods, we can infer that etcd cluster is running well. Could you double check that whether you have network policies enabled which may block the milvus components to directly access the etcd pod ip. Here is the diagnostic procedures:

get etcd pod ip

kubectl get pods -o wide | grep my-release-etcd

login to milvus component pods, for example milvus proxy, check the port connectivity for etcd

kubectl exec -it <milvus-proxy-pod> -- bash
nc -zv <etcd-pod-ip> 2379

if milvus components can access etcd pod with specified port, then everything is ok; if not, you'd check the network policies like this:

kubectl get networkpolicies

if you found networkpolicies, try to remove otherwise you may need contact your gke cluster administrator.
Note: if nc command is not found, you'd install nc like this: apt update && apt install netcat

matsuo1234567 Sep 9, 2024
Author

Upon checking, the network policy was not enabled and the connection was confirmed in all etcd pods.

% kubectl get pods -o wide | grep my-release-etcd
my-release-etcd-0                              1/1     Running     0               3d20h   10.104.77.165   gke-milvus-distribut-milvus-distribut-89938ced-2m7h   <none>           <none>
my-release-etcd-1                              1/1     Running     0               3d20h   10.104.73.24    gke-milvus-distribut-milvus-distribut-89938ced-z7po   <none>           <none>
my-release-etcd-2                              1/1     Running     0               3d20h   10.104.77.161   gke-milvus-distribut-milvus-distribut-89938ced-2m7h   <none>           <none>

% kubectl exec -it my-release-milvus-proxy-65596c894d-fdtzv -- bash
root@my-release-milvus-proxy-65596c894d-fdtzv:/milvus# nc -zv 10.104.77.165 2379
Connection to 10.104.77.165 2379 port [tcp/*] succeeded!
root@my-release-milvus-proxy-65596c894d-fdtzv:/milvus# nc -zv 10.104.73.24 2379
Connection to 10.104.73.24 2379 port [tcp/*] succeeded!
root@my-release-milvus-proxy-65596c894d-fdtzv:/milvus# nc -zv 10.104.77.161 2379
Connection to 10.104.77.161 2379 port [tcp/*] succeeded!

% kubectl get networkpolicies                                      
No resources found in default namespace.

LoveEachDay · 2024-09-06T09:01:21Z

LoveEachDay
Sep 6, 2024
Maintainer

Using the following script to paste all the logs.
https://github.com/milvus-io/milvus/blob/master/deployments/export-log/export-milvus-log.sh

4 replies

matsuo1234567 Sep 7, 2024
Author

LoveEachDay Sep 11, 2024
Maintainer

The log you've attached is the output for kubectl describe <pod>. This is not we need. Please attach the real pod log.

matsuo1234567 Sep 15, 2024
Author

I see.
my-release-etcd-0.log
my-release-etcd-1.log
my-release-etcd-2.log
my-release-milvus-datanode-9584c4488-422rc.log
my-release-milvus-indexnode-5f5dd469dc-lpnlg.log
my-release-milvus-mixcoord-867bb7d6fd-mmgmt.log
my-release-milvus-proxy-65596c894d-fdtzv.log
my-release-milvus-querynode-845984f79c-6wfwq.log
my-release-pulsar-bookie-0.log
my-release-pulsar-bookie-1.log
my-release-pulsar-bookie-2.log
my-release-pulsar-broker-0.log
my-release-pulsar-proxy-0.log
my-release-pulsar-recovery-0.log
my-release-pulsar-zookeeper-0.log
my-release-pulsar-zookeeper-1.log
my-release-pulsar-zookeeper-2.log

yhmo Sep 18, 2024
Collaborator

@matsuo1234567
The final log time of querynode is 2024/09/05. The final log time of other components is larger than 2024/09/10. Looks like the querynode is not alive.

matsuo1234567 · 2024-09-11T02:24:59Z

matsuo1234567
Sep 11, 2024
Author

Error rootcoord

Since the rootcoord was not automatically created, rootcoord.yaml is created.

kind: Deployment
metadata:
  name: my-release-rootcoord
  namespace: default
  labels:
    app: rootcoord
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rootcoord
  template:
    metadata:
      labels:
        app: rootcoord
    spec:
      containers:
      - name: rootcoord
        image: milvusdb/milvus:v2.4.10
        ports:
        - containerPort: 9091
        command: ["/bin/sh"] 
        args: ["-c", "milvus run rootcoord"] 


---
apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/app-protocols: '{"milvus":"HTTP2"}'
    cloud.google.com/backend-config: '{"default": "my-release-backendconfig"}'
    cloud.google.com/neg: '{"ingress": true}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"19530":"k8s1-8a08b012-default-my-release-milvus-19530-1a635530"},"zones":["asia-northeast1-a"]}'
    meta.helm.sh/release-name: my-release
    meta.helm.sh/release-namespace: default
  labels:
    app.kubernetes.io/instance: my-release
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: milvus
    app.kubernetes.io/version: 2.4.10
    component: rootcoord
    helm.sh/chart: milvus-4.2.9
  name: my-release-rootcoord
  namespace: default
spec:
  type: ClusterIP  
  selector:
    app: rootcoord
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: milvus
    port: 19530
    protocol: TCP
    targetPort: 19530
  - name: metrics
    port: 9091
    protocol: TCP
    targetPort: 9091
  sessionAffinity: None

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to connect to Milvus deployed on a GKE cluster #36002

{{title}}

Replies: 5 comments 16 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Unable to connect to Milvus deployed on a GKE cluster #36002

matsuo1234567 Sep 5, 2024

Replies: 5 comments · 16 replies

yhmo Sep 5, 2024 Collaborator

matsuo1234567 Sep 5, 2024 Author

yhmo Sep 5, 2024 Collaborator

matsuo1234567 Sep 5, 2024 Author

xiaofan-luan Sep 5, 2024 Maintainer

xiaofan-luan Sep 5, 2024 Maintainer

matsuo1234567 Sep 5, 2024 Author

xiaofan-luan Sep 6, 2024 Maintainer

matsuo1234567 Sep 6, 2024 Author

matsuo1234567 Sep 6, 2024 Author

matsuo1234567 Sep 6, 2024 Author

LoveEachDay Sep 6, 2024 Maintainer

matsuo1234567 Sep 9, 2024 Author

LoveEachDay Sep 9, 2024 Maintainer

matsuo1234567 Sep 9, 2024 Author

LoveEachDay Sep 6, 2024 Maintainer

matsuo1234567 Sep 7, 2024 Author

LoveEachDay Sep 11, 2024 Maintainer

matsuo1234567 Sep 15, 2024 Author

yhmo Sep 18, 2024 Collaborator

matsuo1234567 Sep 11, 2024 Author

matsuo1234567
Sep 5, 2024

Replies: 5 comments 16 replies

yhmo
Sep 5, 2024
Collaborator

matsuo1234567 Sep 5, 2024
Author

yhmo Sep 5, 2024
Collaborator

matsuo1234567 Sep 5, 2024
Author

xiaofan-luan
Sep 5, 2024
Maintainer

xiaofan-luan Sep 5, 2024
Maintainer

matsuo1234567 Sep 5, 2024
Author

xiaofan-luan Sep 6, 2024
Maintainer

matsuo1234567 Sep 6, 2024
Author

matsuo1234567 Sep 6, 2024
Author

matsuo1234567
Sep 6, 2024
Author

LoveEachDay Sep 6, 2024
Maintainer

matsuo1234567 Sep 9, 2024
Author

LoveEachDay Sep 9, 2024
Maintainer

matsuo1234567 Sep 9, 2024
Author

LoveEachDay
Sep 6, 2024
Maintainer

matsuo1234567 Sep 7, 2024
Author

LoveEachDay Sep 11, 2024
Maintainer

matsuo1234567 Sep 15, 2024
Author

yhmo Sep 18, 2024
Collaborator

matsuo1234567
Sep 11, 2024
Author