Getting a unflushed vector by ID and explicitly not flushing it

[gigatesseract]

Hi folks, not sure if its repeated, couldn't find any so far. I will keep looking.

My use case:

• I want to insert a vector into a collection. Lets call is V1.
• I want to perform similarity search on that vector (similarity on V1)
• I don't want that vector to be a part of similarity search for any other vector. (V1 should not be used for any other vector's similarity search)

I am solving this by application logic. We do the recommended auto flush vs flush based on data ingestion rates. Flushing takes time and I want to see if I can eliminate that.

Here's my understanding of flush-

• Milvus first writes in an in-memory store. If the entities reach a certain threshold / upon manual signal, they are persisted permanently.

My question:

• Is the in-memory store searchable?
  • Does it search only in the in-memory store or also in the db
  • How do we instruct Milvus to restrict its searching / querying only on the in-memory store?
• Is the in-memory store indexable? (Can I get the vector by ID?)
• Can we exclude certain items from being flushed automatically? (I understand the onus would be on the admin to clear them from the in-memory cache)

[yhmo]

As we know, Milvus cluster includes a Pulsar(or Kafka) as a message queue which is like a WAL component. In Milvus standalone, the Pulsar/Kafka is replaced by a built-in RocksDB.

In milvus, search() is for ANN similarity search, query() is for data retrieval such as get vector by id.

When you call collection.insert(data=[V1]) from client side:

1. The proxy node receives the request and passes the request to the message queue. Data in this message queue is not searchable.
2. The query node consumes the request from the message queue asynchronously. The vector is accumulated in a memory buffer, we can call it "growing segment". Data in the memory buffer is searchable/queryable, data visibility depends on the consistency level. If the consistency level is Bounded or Eventually, the data might be invisible to search/query. If the consistency is Strong, the search/query will wait until the latest data is consumed so that it is visible.
3. The memory buffer is mutable. Query node also loads sealed segments into memory, each seal segment is a constant data block. Search is applied both on the mutable memory buffer and the in-memory sealed segments. You can't instruct Milvus to restrict search/query only on the mutable buffer. But you can instruct Milvus to skip search/query on the mutable buffer by specify a "ignore_growing" parameter to search().

The mutable memory buffer can be indexed, the configuration in milvus.yaml, it is a IVF_FLAT index: https://milvus.io/docs/configure_querynode.md#queryNodesegcoreInterimIndex

queryNode:
  segcore:
    interimIndex:
      enableIndex: true # Enable segment build with index to accelerate vector search when segment is in growing or binlog.
      nlist: 128 # temp index nlist, recommend to set sqrt(chunkRows), must smaller than chunkRows/8
      nprobe: 16 # nprobe to search small index, based on your accuracy requirement, must smaller than nlist

We don't recommend users to call flush() manually because frequent flush action will generate lots of tiny segments and bring heavy impact to compaction machinery.
If you want to delete an item, you should use collection.delete(expr="id =xxx"). A delete request is also passed to the message queue, and consumed by query node asynchronously. Once the delete request is consumed by query node, the item will not be searchable/queryable.

[xiaofan-luan]

Hi folks, not sure if its repeated, couldn't find any so far. I will keep looking.

My use case:

• I want to insert a vector into a collection. Lets call is V1.
• I want to perform similarity search on that vector (similarity on V1)
• I don't want that vector to be a part of similarity search for any other vector. (V1 should not be used for any other vector's similarity search)

I am solving this by application logic. We do the recommended auto flush vs flush based on data ingestion rates. Flushing takes time and I want to see if I can eliminate that.

Here's my understanding of flush-

• Milvus first writes in an in-memory store. If the entities reach a certain threshold / upon manual signal, they are persisted permanently.

My question:

• Is the in-memory store searchable?

  • Does it search only in the in-memory store or also in the db
  • How do we instruct Milvus to restrict its searching / querying only on the in-memory store?

• Is the in-memory store indexable? (Can I get the vector by ID?)

• Can we exclude certain items from being flushed automatically? (I understand the onus would be on the admin to clear them from the in-memory cache)

I want to insert a vector into a collection. Lets call is V1.
I want to perform similarity search on that vector (similarity on V1)
I don't want that vector to be a part of similarity search for any other vector. (V1 should not be used for any other vector's similarity search)

is this a scalar filtering requirement?