Bulk Insert and segment size

[ronbarhen]

Hi,
For some justified reasons, I want my segment size to be about 18GB.
I use bulk_insert() in order to ingest a database of 500 million 96-float32 features.
But it seems that the size of the segment is determined according to the size of each numpy file sent during the bulk insert process (which is limited to 4GB).
How can I manage to have a segment size of 18GB?
Should I use another method in order to ingest the data so that my segment size is according to the value set in milvus.yaml?
Thanks!

[yhmo]

Segment size is defined by the milvus.yaml, not bulk_insert().
The dataCoord.segment.maxSize defines the expected size of each segment. In runtime, due to compaction behavior, segment size could be smaller or larger than this value. Anyway, most of segments should be near this value.

dataCoord:
  segment:
    maxSize: 1024 # Maximum size of a segment in MB

bulk_insert() splits the data according to this value. Assume a numpy file size is 10GB, bulk_insert() will split it into 10 segments with 1GB size per segment.

If you want a 18GB segment. You can set dataCoord.segment.maxSize to be 18432, and use a huge numpy file(18GB) to try.

[ronbarhen]

There is apparently a limit on the size of the numpy file.
I get the following error:
The task 452632801283672905 failed, reason: The import file size has reached the maximum limit allowed for importing, fileSize=19599515352, maxSize=17179869184

[ronbarhen]

The size all the numpy files for a given task, including the metadata! is limited to 16GB.
So, I cannot generate segments of 18GB with bulk_insert().
Is there another way?

[yhmo]

Change the configuration in the milvus.yaml, set dataNode.import.maxImportFileSizeInGB to 20.

dataNode:
  import:
    maxImportFileSizeInGB: 20

[yhmo]

Or change this value dataCoord.import.maxSizeInMBPerImportTask to 20000.

dataCoord:
  import:
    maxSizeInMBPerImportTask: 20000   # To prevent generating of small segments, we will re-group imported files. This parameter represents the sum of file sizes in each group (each ImportTask).

You don't have to change the dataNode.import.maxImportFileSizeInGB. Just input multiple files into bulk_insert(), make sure each file size is smaller than 16GB, and make sure the total size of each task is smaller than maxSizeInMBPerImportTask.
One call of bulk_insert() is one importTask.

[xiaofan-luan]

Hi, For some justified reasons, I want my segment size to be about 18GB. I use bulk_insert() in order to ingest a database of 500 million 96-float32 features. But it seems that the size of the segment is determined according to the size of each numpy file sent during the bulk insert process (which is limited to 4GB). How can I manage to have a segment size of 18GB? Should I use another method in order to ingest the data so that my segment size is according to the value set in milvus.yaml? Thanks!

Once you set the segemnt size to larger, compaction will help to merge small segment to larger.
However, I don't recommend to set segment size to that large(18GB). Usually 8GB is large enough for most use cases