Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sections code for vector search #166

Merged
merged 29 commits into from
Nov 2, 2023
Merged

Sections code for vector search #166

merged 29 commits into from
Nov 2, 2023

Conversation

Thejas-bhat
Copy link
Member

@Thejas-bhat Thejas-bhat commented Jul 31, 2023

  • Introduces a new section called faissVectorIndexSection which maintains vector search related operations over a specific field - indexing a bunch of vectors and performing the similarity search on them.
  • The section implements the section interface from section.go which basically ensures that the vector indexing related stuff that happens during the segment's lifecycle is agnostic of any other section that's present (for eg inverted index for textual data).
  • The section itself uses the https://github.com/blevesearch/faiss (a fork from the facebookreasearch's faiss library) library to do the actual work of building the vector index and performing search over it.
  • Currently, two types of index - flat or IVFSQ is built and maintained based on the volume of vectors present for a field within a segment.
  • Currently during search time, the underlying faiss index is consulted to get the similar vectors, constructs a vector postings list which is iterable and each element within it is a pair of docNum in which the similar vector is present it and the corresponding vector's score.
  • Caution: Still under active development.

section_vector_index.go Outdated Show resolved Hide resolved
new.go Outdated Show resolved Hide resolved
section_vector_index.go Outdated Show resolved Hide resolved
section_vector_index.go Outdated Show resolved Hide resolved
section_vector_index.go Outdated Show resolved Hide resolved
go.mod Outdated Show resolved Hide resolved
Copy link
Member

@abhinavdangeti abhinavdangeti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking forward to a lot of commentary here :)

section_vector_index.go Outdated Show resolved Hide resolved
abhinavdangeti and others added 9 commits November 1, 2023 11:38
* updating merge path with the ivf index creations

* Fixes to API usage, etc.

* bug fix: using the correct APIs to get ivf sub-index

* Fix refactored API reference

* Update go.mod to point to select versions

- bleve_index_api,
- scorch_segment_api,
- go_faiss

* Update GO VER and fix workflows

---------

Co-authored-by: Thejas-bhat <[email protected]>
+ Also, in registerSegmentSection - forgot to unlock :D
@Thejas-bhat Thejas-bhat changed the title sections code for DVS Sections code for vector search Nov 2, 2023
@abhinavdangeti abhinavdangeti merged commit 6ed2ef3 into refactor-section Nov 2, 2023
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants