Zookeeper probes fail

[caroline-costa-sn]

Hello,

I have a Milvus cluster deployment on GKE using pulsar. We keep getting these warnings that the Zookeeper liveness and readiness probes fail with the message "ZK server is not OK". It doesn't seem like a big problem, but the messages are flooding our incident reports in GCP. We tried to increase the time limit of the probes, which reduced the number of warnings, but has not fixed the problem. Does anyone have any other suggestions?

[yhmo]

Pulsar is a third-party independent component for Milvus, I think it might be a deployment issue or related to the host environment.
A similar issue: #16062

[caroline-costa-sn]

Thanks @yhmo. Yes, I found that discussion and tried the suggestions there, but it did not fix the issue.

[njcolvin]

Hi. Did you ever determine how to fix this issue? Experiencing the same thing. New to kubernetes and not sure how to debug zookeeper for deployment on GCP. I tried increasing the period and timeout, but the errors still occur frequently.

[xiaofan-luan]

@njcolvin
if you are more familiar with kafka maybe you should try to deploy kafka.
zk usually a very stable compomnent. you probabl want to check all 3 zk are alive and they are running with enough disk space

[njcolvin]

Thanks for the reply. Unfortunately I'm not familiar with kafka either. Perhaps it is a resource issue as you hinted.

Speaking of resources, I am a bit confused by the cluster sizing tool. It estimates I need 162 cores and 553GB memory to host 10M 1536 dim vectors (segment size 1024MB). This seems like a lot. A GCP machine with those resources (c3-standard-176) costs $6500 per month. Is my understanding correct? My guess is at least one of the following must be true:

1. The cluster sizing tool estimate is conservative
2. A c3-standard-176 machine isn't as expensive in practice with proper management, or
3. Zilliz Cloud is more cost-effective than deploying a GKE cluster (the same configuration with the Zilliz Cloud pricing calculator estimates $1580 per month for performance-optimized and $396 for capacity-optimized)

I'd appreciate if someone could clear this up for me. Thanks.

[xiaofan-luan]

Hi @njcolvin
You have several way to save your cost:

1. use standalone cluster. with standalone cluster, you only need around 16core 128GB to host 10 million 1536 data(rough guess)
2. the distributed deployment requires many extra dependencies，you can cut most of the dependency to half size.
3. Zilliz cloud can definitely help you to save. For perf instance, 10m 1536 takes around 16cu(~1800 a month), while capacity instance only need 4cu(450 a month). That is because we use special quantization on the cloud and using disk to offload original data.