Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Onboard hybrid search use case; add readonly search flows for all use cases #143

Merged
merged 6 commits into from
Apr 22, 2024

Conversation

ohltyler
Copy link
Member

@ohltyler ohltyler commented Apr 22, 2024

Context

This PR adds many changes to the I/O and appearance on the drag-and-drop workspace. Note the final inputs/outputs/component types etc. are still not finalized. The purpose for adding placeholder components and updating inputs/outputs now is to iteratively improve on and make the current state of the plugin more functional and user-friendly, and to help start gathering initial feedback. Adding the existing use cases helps provide some immediate utility and testing of the corresponding backend Flow Framework plugin.

Description

This PR has 2 main changes: 1/ Onboarding a basic hybrid search use case, and 2/ providing visible, readonly flows on the DnD workspace for better overall e2e understanding from user perspective. More details:

  • adds several UI drag-and-drop components for both hybrid search use case (results transformer), and for search flow visibility (several query-related blocks)
  • modifies several existing components to handle inputs for both ingest and search (e.g., all transformers can now accept a document to be used on ingest, or a query to be used on search)
  • adds interfaces and functionality to create search pipelines
  • integrates the new use case into the prototyping components- Ingestor and QueryExecutor
  • makes resource descriptions on resource list tab more robust by determining type based on step type vs. resource type - this is due to a backend limitation of both ingest and search pipelines falling under the same pipeline_id resource type
  • minor refactoring of helper fns to handle more use cases
  • enhances searchIndex() end-to-end flow to process an optional search_pipeline param to propagate to the cluster. this is used behind-the-scenes in the QueryExecutor when executing the searches, if the use case is of type HYBRID_SEARCH.

And some minor tweaks to the reactflow workspace:

  • makes components draggable again, but still readonly
  • dynamically rendering color based on if the component needs input or is just a placeholder

Demo video:

  • selecting new hybrid search option
  • view of the end-to-end flows for both ingest and search (note all search components are readonly currently as we hardcode the results transformer/search pipeline)
  • creating and provisioning all of the resources needed for hybrid search, including a search pipeline
  • executing ingest
  • executing search with a hybrid query template, and using the created search pipeline behind the scenes
screen-capture.28.webm

Screenshots of search flows added for other use cases:
Semantic search:

semantic

Neural sparse search:

neural-sparse

Issues Resolved

Makes progress on #23, #68

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ohltyler ohltyler merged commit 73cbed4 into opensearch-project:main Apr 22, 2024
9 checks passed
@ohltyler ohltyler deleted the hybrid-search branch April 22, 2024 16:08
opensearch-trigger-bot bot pushed a commit that referenced this pull request Apr 22, 2024
… cases (#143)

Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit 73cbed4)
ohltyler added a commit that referenced this pull request Apr 22, 2024
… cases (#143) (#144)

Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit 73cbed4)

Co-authored-by: Tyler Ohlsen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants