Adds Create Ingest Pipeline Step #44
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Joshua Palis <[email protected]>
…ments Signed-off-by: Joshua Palis <[email protected]>
Signed-off-by: Joshua Palis <[email protected]>
…rather than from content Signed-off-by: Joshua Palis <[email protected]>
Signed-off-by: Joshua Palis <[email protected]>
Signed-off-by: Joshua Palis <[email protected]>
…avadoc error Signed-off-by: Joshua Palis <[email protected]>
Codecov Report
@@ Coverage Diff @@
## main #44 +/- ##
============================================
+ Coverage 68.33% 73.87% +5.54%
- Complexity 48 65 +17
============================================
Files 7 8 +1
Lines 180 245 +65
Branches 18 22 +4
============================================
+ Hits 123 181 +58
- Misses 48 54 +6
- Partials 9 10 +1
|
Signed-off-by: Joshua Palis <[email protected]>
dbwiddis
reviewed
Sep 21, 2023
src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java
Outdated
Show resolved
Hide resolved
…kflow data for required fields Signed-off-by: Joshua Palis <[email protected]>
Signed-off-by: Joshua Palis <[email protected]>
Signed-off-by: Joshua Palis <[email protected]>
…st body generator Signed-off-by: Joshua Palis <[email protected]>
joshpalis
requested review from
owaiskazi19,
ohltyler,
amitgalitz and
jackiehanyang
as code owners
Signed-off-by: Joshua Palis <[email protected]>
dbwiddis
reviewed
Sep 22, 2023
src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java
Show resolved
Hide resolved
src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java
Show resolved
Hide resolved
...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java
Outdated
Show resolved
Hide resolved
...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java
Outdated
Show resolved
Hide resolved
...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java
Outdated
Show resolved
Hide resolved
...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java
Outdated
Show resolved
Hide resolved
...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java
Outdated
Show resolved
Hide resolved
...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java
Outdated
Show resolved
Hide resolved
...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Joshua Palis <[email protected]>
This was referenced Sep 25, 2023
dbwiddis
approved these changes
Sep 25, 2023
Signed-off-by: Joshua Palis <[email protected]>
owaiskazi19
reviewed
Sep 25, 2023
src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java
Outdated
Show resolved
Hide resolved
src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java
Outdated
Show resolved
Hide resolved
...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java
Outdated
Show resolved
Hide resolved
...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java
Show resolved
Hide resolved
Signed-off-by: Joshua Palis <[email protected]>
Signed-off-by: Joshua Palis <[email protected]>
owaiskazi19
reviewed
Sep 25, 2023
src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java
Show resolved
Hide resolved
owaiskazi19
approved these changes
Sep 25, 2023
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Sep 25, 2023
* Initial create ingest pipeline implementation Signed-off-by: Joshua Palis <[email protected]> * Adding TODOs for storing response to global context index, fixing comments Signed-off-by: Joshua Palis <[email protected]> * Updating workflowData interface, modifying CreateIngestPipelineStep Signed-off-by: Joshua Palis <[email protected]> * updating workflow data extraction to read pipelineId from parameters rather than from content Signed-off-by: Joshua Palis <[email protected]> * removing unecessary cast Signed-off-by: Joshua Palis <[email protected]> * Pulls all required data from content rather than from params, fixes javadoc error Signed-off-by: Joshua Palis <[email protected]> * fixing comments Signed-off-by: Joshua Palis <[email protected]> * addressing PR comments, adding switch statement to handle parsing workflow data for required fields Signed-off-by: Joshua Palis <[email protected]> * Adding entry import Signed-off-by: Joshua Palis <[email protected]> * fixing comments Signed-off-by: Joshua Palis <[email protected]> * Adds unit tests for create ingest pipeline step, fixes pipeline request body generator Signed-off-by: Joshua Palis <[email protected]> * Adding failure tests Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments Signed-off-by: Joshua Palis <[email protected]> * Addressing PR comments Signed-off-by: Joshua Palis <[email protected]> * Fixing workflow data Signed-off-by: Joshua Palis <[email protected]> --------- Signed-off-by: Joshua Palis <[email protected]> (cherry picked from commit 1d22bee) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
joshpalis
added a commit
that referenced
this pull request
Sep 25, 2023
Adds Create Ingest Pipeline Step (#44) * Initial create ingest pipeline implementation * Adding TODOs for storing response to global context index, fixing comments * Updating workflowData interface, modifying CreateIngestPipelineStep * updating workflow data extraction to read pipelineId from parameters rather than from content * removing unecessary cast * Pulls all required data from content rather than from params, fixes javadoc error * fixing comments * addressing PR comments, adding switch statement to handle parsing workflow data for required fields * Adding entry import * fixing comments * Adds unit tests for create ingest pipeline step, fixes pipeline request body generator * Adding failure tests * Addressing PR comments * Addressing PR comments * Fixing workflow data --------- (cherry picked from commit 1d22bee) Signed-off-by: Joshua Palis <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Joshua Palis <[email protected]>
dbwiddis
added a commit
that referenced
this pull request
Sep 28, 2023
#47) * Add WorkflowStepFactory class Signed-off-by: Daniel Widdis <[email protected]> * Add XContent classes representing Template JSON Signed-off-by: Daniel Widdis <[email protected]> * Add parse methods for the Template XContent Signed-off-by: Daniel Widdis <[email protected]> * Cleanup parsing, javadocs, and demo output Signed-off-by: Daniel Widdis <[email protected]> * Refactor to use field name constants, get tests working again Signed-off-by: Daniel Widdis <[email protected]> * Separate WorkflowNode and ProcessNode functionality Signed-off-by: Daniel Widdis <[email protected]> * Fix demos to align with template field names Signed-off-by: Daniel Widdis <[email protected]> * Add workflow, node, and edge tests Signed-off-by: Daniel Widdis <[email protected]> * Add Template tests Signed-off-by: Daniel Widdis <[email protected]> * Refactor TemplateParser to WorkflowProcessSorter Signed-off-by: Daniel Widdis <[email protected]> * Test exceptional cases Signed-off-by: Daniel Widdis <[email protected]> * Finish up exceptional cases Signed-off-by: Daniel Widdis <[email protected]> * Fix a template field name bug in demo Signed-off-by: Daniel Widdis <[email protected]> * Rebase with #34 Signed-off-by: Daniel Widdis <[email protected]> * Rebase changes from #54 Signed-off-by: Daniel Widdis <[email protected]> * Integrate thread pool executor service Signed-off-by: Daniel Widdis <[email protected]> * Fix flaky ProcessNodeTests by removing orTimeout Signed-off-by: Daniel Widdis <[email protected]> * Rebase and refactor with #44 Signed-off-by: Daniel Widdis <[email protected]> * Fix demos and remove DataDemo Signed-off-by: Daniel Widdis <[email protected]> * Use non-deprecated mapping method for CreateIndexStep Signed-off-by: Daniel Widdis <[email protected]> * Eliminate casting and deprecation warnings on test classes Signed-off-by: Daniel Widdis <[email protected]> * Remove unused/leftover demo class Signed-off-by: Daniel Widdis <[email protected]> * Typo Signed-off-by: Daniel Widdis <[email protected]> * Don't offer steps as an alternative to nodes Signed-off-by: Daniel Widdis <[email protected]> * Move Workflow into package with all the other parsing classes Signed-off-by: Daniel Widdis <[email protected]> * Move process sequencing classes into workflow package Signed-off-by: Daniel Widdis <[email protected]> * Add PipelineProcessor class and XContent parsing, rename package Signed-off-by: Daniel Widdis <[email protected]> --------- Signed-off-by: Daniel Widdis <[email protected]>
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Sep 28, 2023
#47) * Add WorkflowStepFactory class Signed-off-by: Daniel Widdis <[email protected]> * Add XContent classes representing Template JSON Signed-off-by: Daniel Widdis <[email protected]> * Add parse methods for the Template XContent Signed-off-by: Daniel Widdis <[email protected]> * Cleanup parsing, javadocs, and demo output Signed-off-by: Daniel Widdis <[email protected]> * Refactor to use field name constants, get tests working again Signed-off-by: Daniel Widdis <[email protected]> * Separate WorkflowNode and ProcessNode functionality Signed-off-by: Daniel Widdis <[email protected]> * Fix demos to align with template field names Signed-off-by: Daniel Widdis <[email protected]> * Add workflow, node, and edge tests Signed-off-by: Daniel Widdis <[email protected]> * Add Template tests Signed-off-by: Daniel Widdis <[email protected]> * Refactor TemplateParser to WorkflowProcessSorter Signed-off-by: Daniel Widdis <[email protected]> * Test exceptional cases Signed-off-by: Daniel Widdis <[email protected]> * Finish up exceptional cases Signed-off-by: Daniel Widdis <[email protected]> * Fix a template field name bug in demo Signed-off-by: Daniel Widdis <[email protected]> * Rebase with #34 Signed-off-by: Daniel Widdis <[email protected]> * Rebase changes from #54 Signed-off-by: Daniel Widdis <[email protected]> * Integrate thread pool executor service Signed-off-by: Daniel Widdis <[email protected]> * Fix flaky ProcessNodeTests by removing orTimeout Signed-off-by: Daniel Widdis <[email protected]> * Rebase and refactor with #44 Signed-off-by: Daniel Widdis <[email protected]> * Fix demos and remove DataDemo Signed-off-by: Daniel Widdis <[email protected]> * Use non-deprecated mapping method for CreateIndexStep Signed-off-by: Daniel Widdis <[email protected]> * Eliminate casting and deprecation warnings on test classes Signed-off-by: Daniel Widdis <[email protected]> * Remove unused/leftover demo class Signed-off-by: Daniel Widdis <[email protected]> * Typo Signed-off-by: Daniel Widdis <[email protected]> * Don't offer steps as an alternative to nodes Signed-off-by: Daniel Widdis <[email protected]> * Move Workflow into package with all the other parsing classes Signed-off-by: Daniel Widdis <[email protected]> * Move process sequencing classes into workflow package Signed-off-by: Daniel Widdis <[email protected]> * Add PipelineProcessor class and XContent parsing, rename package Signed-off-by: Daniel Widdis <[email protected]> --------- Signed-off-by: Daniel Widdis <[email protected]> (cherry picked from commit 734f9c2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
dbwiddis
pushed a commit
that referenced
this pull request
Sep 28, 2023
…Template Parsing (#60) Add WorkflowStep Factory and implement XContent-based Template Parsing (#47) * Add WorkflowStepFactory class * Add XContent classes representing Template JSON * Add parse methods for the Template XContent * Cleanup parsing, javadocs, and demo output * Refactor to use field name constants, get tests working again * Separate WorkflowNode and ProcessNode functionality * Fix demos to align with template field names * Add workflow, node, and edge tests * Add Template tests * Refactor TemplateParser to WorkflowProcessSorter * Test exceptional cases * Finish up exceptional cases * Fix a template field name bug in demo * Rebase with #34 * Rebase changes from #54 * Integrate thread pool executor service * Fix flaky ProcessNodeTests by removing orTimeout * Rebase and refactor with #44 * Fix demos and remove DataDemo * Use non-deprecated mapping method for CreateIndexStep * Eliminate casting and deprecation warnings on test classes * Remove unused/leftover demo class * Typo * Don't offer steps as an alternative to nodes * Move Workflow into package with all the other parsing classes * Move process sequencing classes into workflow package * Add PipelineProcessor class and XContent parsing, rename package --------- (cherry picked from commit 734f9c2) Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds CreateIngestPipelineStep for workflow set up. In this current state, this step is hardcoded to parse the necessary information from the
WorkflowDatarelated to thetext_embeddingprocessor.Issues Resolved
#24
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.