Does the golang SDK support Sparse Vector search?

[juanandreas]

https://milvus.io/docs/sparse_vector.md

This example only shows pymilvus example. Can we do the same with the go sdk? I think currently the go sdk search only expects a vector of type slice.

[xiaofan-luan]

I think this is under development.
could be ready some time this month

[yhmo]

It seems the go sdk v2.4.0 has already supported sparse vector: https://github.com/milvus-io/milvus-sdk-go/releases/tag/v2.4.0
The release note mentioned:

• enhance: Support Sparse Embedding by @congqixia in enhance: Support Sparse Embedding milvus-sdk-go#684

If you don't know how to create sparse vector, read the test code:
https://github.com/milvus-io/milvus-sdk-go/blob/d346d3ed3898244bf17a324767b220b808c52c25/test/common/utils.go#L1348
https://github.com/milvus-io/milvus-sdk-go/blob/d346d3ed3898244bf17a324767b220b808c52c25/test/common/utils.go#L229
https://github.com/milvus-io/milvus-sdk-go/blob/d346d3ed3898244bf17a324767b220b808c52c25/test/testcases/search_test.go#L1665

[juanandreas]

Correct me if I'm wrong but looking from the code I think it only supports creating sparse vectors, but the client has yet to support SEARCH on collections with sparse vectors.

[juanandreas]

I followed the L1665 test cased in search_test.go. I was able to successfully run a sparse vector search but I am only getting one result. Trying the same inputs and parameters on pymilvus, I get 50 results. Does the search code work the same under the hood for both SDKs?

I see that using client.Query() gets me 50 results. Why does client.Search give one result but client.Query returns many?

[yhmo]

This line generates 5 sparse vectors since the common.DefaultNq=5: https://github.com/milvus-io/milvus-sdk-go/blob/d346d3ed3898244bf17a324767b220b808c52c25/test/testcases/search_test.go#L1690C22-L1690C38

This line verifies that the search result contains 5 groups of topk items: https://github.com/milvus-io/milvus-sdk-go/blob/d346d3ed3898244bf17a324767b220b808c52c25/test/testcases/search_test.go#L1695

This line verifies that each result group has top10 items since the common.DefaultTopK=10: https://github.com/milvus-io/milvus-sdk-go/blob/d346d3ed3898244bf17a324767b220b808c52c25/test/testcases/search_test.go#L1699

The search() interface allows multiple vectors by the "queryVec".
If there are 2 vectors in "queryVec", topk=3, the search result will look like:

{
   "top5 of the first vector": [
        "top1": {id: xxx, distance: xxxx},
        "top2": {id: xxx, distance: xxxx},
        "top3": {id: xxx, distance: xxxx},
   ],
   "top5 of the second vector": [
        "top1": {id: xxx, distance: xxxx},
        "top2": {id: xxx, distance: xxxx},
        "top3": {id: xxx, distance: xxxx},
   ],
}

So, double-check your input and the result. The test case already verified it is correct, I don't think the SDK has any problem.

[juanandreas]

@xiaofan-luan There seems to be a bug in the go SDK vector search.

In the python SDK, when I set my query limit to 50, I get 50 hits in my searchResult object that gets returned from vector search:

search_param = {
  "data": [query_vector],
  "anns_field": "sparse_vector_column",
  "param": {"metric_type": "IP", "params":{"drop_ratio_search": 0.2}},
  "limit": 50,
  "expr": f"id=='{id}'",
  "output_fields": ["id", "secondary_id_array_column", "sparse_vector_column"]
}

In the above query, I am getting 50 results of maybe 1 or 2 strings in secondary_id_array_column field and a short sparse vector for the sparse_vector_column field.

But in go SDK, for the same exact query, my results seem to all be getting mashed together. The GO SDK vector search returns a []SearchResult correct? But then when I tries getting the length of []SearchResult, I only see that it is of length 1 and all of my "secondary_id_array_column" & "sparse_vector_column" values seems to be getting combined. In this case, I am getting 55 values in my secondary_id_array_column field and a longer sparse vector for the sparse_vector_column field.

	inputSparseEmbedding, err := entity.NewSliceSparseEmbedding(positions, values)

	queryVec := make([]entity.Vector, 0, common.DefaultNq)
	queryVec = append(queryVec, inputSparseEmbedding)

        topK := 50

	searchResults, err := milvusClient.Search(
		context.Background(), // ctx
		CollectionName, // collName
		[]string{},       // partitions
		attributeFilters, // expr
		[]string{"*"},    //outputFields
		queryVec,         // vectors
		"relevance",      // vectorField
		entity.IP,        // metricType
		topK,             // topK
		sp,               // sp
	)

Please also see my bug ticket about being unable to get the sparse vector column value in the GO SDK.
milvus-io/milvus-sdk-go#791

[xiaofan-luan]

@aoiasd
can you double confirm this

[juanandreas]

@xiaofan-luan I discovered that we need to retrieve the searchResult hits via

hits.Fields.GetColumn("col_name").Slice(0, <some_integer>).FieldData().GetScalars().GetArrayData().Data

or, for sparse vector values from the hits:

hits.Fields.GetColumn("sparse_vector_col_name").Slice(0, <some_integer>).FieldData().GetVectors().GetSparseFloatVector().GetContents()

However, the above sparse vector content retrieval gives us []byte encrypted sparse vectors. I had to implement a base64 decoder to convert it to map[int]float32. Is there an easier way to do this without having to decode or is it a service level agreement product decision? I believe we do not receive encrypted sparse vector values with the python sdk.

Is there also an easier way to retrieve values without having to make so many calls? What was the decision for the lengthy series of function calls just to get searchResult hits values?

[aoiasd]

@xiaofan-luan I discovered that we need to retrieve the searchResult hits via

hits.Fields.GetColumn("col_name").Slice(0, <some_integer>).FieldData().GetScalars().GetArrayData().Data

or, for sparse vector values from the hits:

hits.Fields.GetColumn("sparse_vector_col_name").Slice(0, <some_integer>).FieldData().GetVectors().GetSparseFloatVector().GetContents()

However, the above sparse vector content retrieval gives us []byte encrypted sparse vectors. I had to implement a base64 decoder to convert it to map[int]float32. Is there an easier way to do this without having to decode or is it a service level agreement product decision? I believe we do not receive encrypted sparse vector values with the python sdk.

Is there also an easier way to retrieve values without having to make so many calls? What was the decision for the lengthy series of function calls just to get searchResult hits values?

FieldData() used to get result proto data，so you will get the source data like []byte

Usually we could get colum data like
hits.Fields.GetColumn("pk").GetAsInt64(id)
but now we seems lack GetAsSparseVector.

Maybe you can get sparse data by

	vector, err := hits.Fields.GetColumn("vector").Get(id)
	if err != nil {
		return err
	}
	sparse,ok:= vector.(entity.SparseEmbedding)

or

	vectorColumn := hits.Fields.GetColumn("vector").(*entity.ColumnSparseFloatVector)
	sparse, err := vectorColumn.ValueByIdx(idx)
	if err != nil {
		return err
	}

But now SparseEmbedding only support get position and value by index, not return a map.

Seems we should complete GetSparseVector for Column and add Mapping method for SparseEmbedding.

[xiaofan-luan]

@aoiasd
please help on implementing this