When inserting a set of data, an error occurred: timeout

[pincutionhot]

In a Milvus cluster, I have a large collection with about 200 million rows of data. I had build the index, and the collection is not loaded. I need to add another 4000 million rows to it. However, I often encounter timeouts when inserting, which stops my data import program, and it will be normal for a while after stopping. I need to continue to import about 600K data per hour. Are there any tips to speed up the whole process?

[xiaofan-luan]

can you offer logs of your cluster?
4B seems to be a huge cluster, how many resources do you have in a cluster?
How many querynodes, datanodes and index nodes?
feel free to contact me at [email protected] and I'd like to learn more about your use cases

[pincutionhot]

I apologize but I cannot provide you with the logs today.
The milvus's version is 2.3.18, index type: ivf_sq8, dim: 512
I have six hosts, each with 1TB of memory, two 32-core CPUs, and 8TB SSD.
I have already expanded the indexnodes's replicas to 6 and the datanodes to 3. The collection is not loaded, querynode's replicas is 1. The cluster's storage is managed by Rook-Ceph.
Should I provide logs from a specific pod, or would it be better to deploy a dedicated log pod for centralized log collection?

[xiaofan-luan]

I apologize but I cannot provide you with the logs today. The milvus's version is 2.3.18, index type: ivf_sq8, dim: 512 I have six hosts, each with 1TB of memory, two 32-core CPUs, and 8TB SSD. I have already expanded the indexnodes's replicas to 6 and the datanodes to 3. The collection is not loaded, querynode's replicas is 1. The cluster's storage is managed by Rook-Ceph. Should I provide logs from a specific pod, or would it be better to deploy a dedicated log pod for centralized log collection?

you have to find which node crashed and collect it's log.

also check quota limitation see if any quota limitation is triggered

[xiaofan-luan]

could be due to lack to disk space or time tick delay