-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reciprocal Rank Fusion (RRF) normalization technique in hybrid query #874
base: feature/rrf-score-normalization-v2
Are you sure you want to change the base?
Reciprocal Rank Fusion (RRF) normalization technique in hybrid query #874
Conversation
Signed-off-by: Isaac Johnson <[email protected]>
Signed-off-by: Isaac Johnson <[email protected]>
Signed-off-by: Isaac Johnson <[email protected]>
Signed-off-by: Isaac Johnson <[email protected]>
Signed-off-by: Isaac Johnson <[email protected]>
we should be merging to feature branch https://github.com/opensearch-project/neural-search/tree/feature/rrf-score-normalization, not main. |
List<TopDocs> topDocsPerSubQuery = compoundQueryTopDocs.getTopDocs(); | ||
int numSubQueriesBound = topDocsPerSubQuery.size(); | ||
for (int index = 0; index < numSubQueriesBound; index++) { | ||
int numDocsPerSubQueryBound = topDocsPerSubQuery.get(index).scoreDocs.length; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can have score docs array as a variable, should save few cpu cycles.
ScoreDoc[] scoreDocs = topDocsPerSubQuery.get(index).scoreDocs;
@NonNull | ||
private ScoreCombinationTechnique combinationTechnique; | ||
@Nullable | ||
private int rankConstant; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the rank constant should be a parameter for RRF normalization technique class, not the high level field in this DTO. You can pass it as param to RRF normalization technique constructor while creating its instance in a factory class
Description
Adding ability to process and combine scores from multiple subqueries in neural search using the reciprocal rank fusion (RRF) technique. Built with a new processor and processor factory class apart from NormalizationProcessor. Changes to API included in RFC. Does not currently support weights when combining processed subquery scores, based on lack of examples in existing literature.
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
#865
#659
Check List
--signoff
.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.