Handle zero chunk size #264
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* index sections * faiss index section + inverted text index section Co-authored-by: Marty Schoch <[email protected]> Co-authored-by: Abhi Dangeti <[email protected]>
* index.Delete() to free up memory * refactoring index creation for IVF type of indexes * absorbing API changes from go-faiss
cleaning up index.Close() errors
To absorb: * 0ea762e Abhi Dangeti | Temporarily revert free-ing C pointers/buffers - adding a TODO for this (#7)
* e5f7515 Thejas-bhat | bug fix: copying the serialized content from C heap to go (#8)
…document (#181) * avoiding bitmaps when the vector is present only in 1 doc * minor refactor of the code * code comment
Co-authored-by: Abhi Dangeti <[email protected]>
* using IVF instead of flat for smaller indexes * unit test fix * cleanup; bug fix: track erro from add_with_ids
* command line support for vector section - initial commit * docvalue tool fixes * bug fix: docvalue cmd returning nil result * refactoring fields cmd tool * code cleanup
* choosing the index types more responsibly * code cleanup; tests fix
* bug fix: correcting the valid vector ids being tracked during merge * fixing the decoding segment data in vector cmd tool * bug fix: shift the docNum by 32 to account for signed score vals
* bug fix: correcting the valid vector ids being tracked during merge * fixing the decoding segment data in vector cmd tool * bug fix: shift the docNum by 32 to account for signed score vals * resetting vec datastructures on a opaque.Reset() * tracking the number of vecs for future allocations * allocateSpace -> realloc * Update section_faiss_vector_index.go, segment.go --------- Co-authored-by: Abhi Dangeti <[email protected]>
* separate functions to read and search vector indexes. * minor update * addressed reviews * common func for doc values
…ed (#196) bug fixes: using seg.ErrClosed when the merge is aborted
…type. (#197) * switch to array when deleting vectors * minor UT improvements * redo the merge operation * comment and UT fixes * added back reconstruction method for special cases * minor improvements * fix just for IVF indexes * snake case to camel case * train only IVF indexes, not flat
* support nested/chunked vectors * improve commentary * address review comments * Conflict resolution --------- Co-authored-by: Abhinav Dangeti <[email protected]>
bug fix: fixing out of bound slice errors
…199) * refactoring vector index metadata tracking * tracking only the relevant segments while merging * minor code cleanup
dropping the unnecessary masking bits; unit test fix
* reconstruction for all index types * correct ordering of vector IDs * remove unused code * fixed commentary
* changing defaults for nlist and nprobe * address review * addressed reviews * better func naming * Minor optimizations, fix naming, upgrade to blevesearch/[email protected] --------- Co-authored-by: Abhinav Dangeti <[email protected]>
…ch (#205) This PR addresses filtering of deleted documents by moving from post-filtering to search time filtering. Passes doc IDs to exclude for each search. To account for duplicate vectors: saves every vector with a unique hash so that the selector only excludes the specific vector ID. The current approach of identical vectors having identical hashes would lead to (unnecessarily) excluding vectors if other (identical) vectors in the segment were deleted. * Upgrade to [email protected] --------- Co-authored-by: Abhinav Dangeti <[email protected]>
- accommodates user choice of optimising for speed/recall when creating an index - it does this by changing the nprobe count based on the index type. - persists this so that this is adhered to during the merge process as well. --------- Co-authored-by: Abhinav Dangeti <[email protected]>
* when the inverted index section processes the vector fields as well, there are some amount of allocations possible for content which its not supposed to process (for example vector content. This PR fixes that particular short-sight. * during the merge process for every field we are currently allocating a couple of slices of capacity = number of segments. these allocations can be reused * currently the freeing of the reconstructed vector indexes (which can be large in high volume) is a defer logic. This holds up the memory on C heap until the end of function before which there are expensive steps of train and add to the merged index - which will be detrimental overall particularly while introducing the segments as well in high data ingestion scenarios.
+ This change will hugely save compute in iterating through a potentially massive vecDocIDMap on every search call. + Pivoting to populating vectorIDsToExclude during the segment interpret operation - just the one time where we build the vecDocIDMap. Requires: blevesearch/scorch_segment_api#40
add coarse quantiser
- Uses the exponentially weighted moving average to get the average hits on a particular field (thereby a particular vector index). The index stays in the memory when the average of the hits is above a certain threshold and below which the index is closed and the memory is freed for reuse on C side of things. - This ensures that we are not keeping the index in the memory even when there is no query workload on the field in a segment so the memory pressure does get reduced on the C side of operations. --------- Co-authored-by: Likith B <[email protected]> Co-authored-by: Abhinav Dangeti <[email protected]>
* minor optimizations and bug fixes * resolve comment
- Generalised some of the cache function names to be inclusive of the map - Added the map to the cache which will behave the same as the index - Except bitmap logic is not part of the cache and the vecs excluded is calculated outside of the map --------- Co-authored-by: Abhinav Dangeti <[email protected]>
To include: * 693b06a Rahul Rampure | MB-61650: Release IDSelectorBatch's batchSelector to avoid memory leak
Includes: * d8f2ddf Abhi Dangeti | MB-60697: Windows requires nprobe to be of 'long long' type (#24) * 9bb55f8 Abhi Dangeti | Retain IDSelector's Delete() API for complete-ness
- Defer the stopping of the monitor routine's ticker to release the ticker's resources
Requires: - blevesearch/faiss#17 - blevesearch/go-faiss#26 (Upgrade to blevesearch/[email protected]) --------- Co-authored-by: Abhinav Dangeti <[email protected]>
bug fix: de-duplicating fieldsInv entry for a segment
* updated quantiser * Upgrade bleve_index_api, scorch_segment_api Brings in: * f4827a8 Aditi Ahuja | MB-60943 - Add option for memory-optimised vector indexes. --------- Co-authored-by: Abhinav Dangeti <[email protected]>
Refactors IndexOptimizedForMemory -> IndexOptimizedForMemoryEfficient
* MB-62167: Fix windows crash * vectorIndexIOFlags -> faissIOFlags
…of OMP Threads to be 1 (#247) * Use the go-faiss OpenMP API for setting the default number of OMP Threads to be 1 * Upgrade to [email protected] * ec45499 Rahul Rampure | MB-61930: Improve threading performance --------- Co-authored-by: Abhinav Dangeti <[email protected]>
Brings in: * 7531ec8 Rahul Rampure | MB-62221: Fix platform specific behaviour
* MB-61889: support search with params * Upgrade to [email protected] & [email protected] * Upgrade to blevesearch/[email protected] * 88dd5e2 Mohd Shaad Khan | Mb 61889: support search with params --------- Co-authored-by: Abhinav Dangeti <[email protected]>
#252) * handling buffer overflow while creating new segmentBase * cleaning up code instrumentation * more comments around the edge case
* Set FAISS vector metric to be `MetricInnerProduct`, if the distance metric from bleve is either `Cosine` or `InnerProduct` * Upgrade to [email protected] --------- Co-authored-by: Abhinav Dangeti <[email protected]>
* 1bb080b Abhi Dangeti | MB-61889: Address corner case with ivf_nprobe_pct
*Accommodates filtered doc IDs, if required, within a kNN search over a vector index. * Builds and caches a document to vector ID map for looking up vector IDs of the filtered doc IDs. * Account for nested vectors * Upgrade bleve_index_api, scorch_segment_api, go-faiss & workflows --------- Co-authored-by: Abhinav Dangeti <[email protected]>
The existing implementation of the method getChunkSize(...) (uint64, error), in some cases, can return chunkSize==0 and err==nil. The onus is on the caller to check for such possibility and handle it properly. Callers often use the returned chunkSize as a divisor and a zero chunkSize lead to panic. see #209 This PR intends to update the method implementation to always return an error in case the returned chunkSize value is 0. That way caller need to only worry about error being non-nil. Callers which are ok with 0 chunkSize can check the returned error against ErrChunkSizeZero
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of #263