Adding create ingest pipeline step

[amitgalitz]

Description

Created ingest pipeline step so we can create a pipeline through a template. This supports any number of processors, and we support all processors here due to the limited validation. The pipeline steps also supports taking in a model ID currently from a previous step through substitution.

The two required fields for a pipeline are the pipelineId which is also referred to as pipeline name, and configurations which refer to the configurations of the pipeline, this includes the processors list, the description, tag or other optional lists. In order to support any ingest processor out there without ammending this step each time a new one is added, we convert the configurations given to a string and pass that string without extra validation on our end to the create pipeline API.

Examples:

                {
                    "id": "create_ingest_pipeline",
                    "type": "create_ingest_pipeline",
                    "previous_node_inputs": {
                        "deploy_openai_model": "model_id"
                    },
                    "user_inputs": {
                        "pipeline_id": "text-embedding-3",
                        "configurations": {
                            "description": "A text embedding pipeline",
                            "processors": [
                                {
                                    "text_embedding": {
                                        "model_id": "${{deploy_openai_model.model_id}}",
                                        "field_map": {
                                            "passage_text": "passage_embedding"
                                        }
                                    }
                                }
                            ]
                        }
                    }

                {
                  "id": "create_ingest_pipeline_2",
                  "type": "create_ingest_pipeline",
                  "previous_node_inputs": {
                      "deploy_openai_model": "model_id"
                  },
                  "user_inputs": {
                      "pipeline_id": "append-1",
                      "configurations": {
                          "description": "Pipeline that appends event type",
                          "processors": [
                              {
                                  "append": {
                                      "field": "event_types",
                                      "value": [
                                          "page_view"
                                      ]
                                  }
                              },
                              {
                                  "drop": {
                                      "if": "ctx.user_info.contains('password') || ctx.user_info.contains('credit card')"
                                  }
                              }
                          ]
                      }
                  }
              }

Issues Resolved

resolves #24

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[joshpalis]

Overall looks good to me, just a few comments

[dbwiddis]

Not sure how much of this is working around the bug fixed in #559, but can definitely improve the parsing...

[codecov]

Codecov Report

Attention: Patch coverage is 67.34694% with 16 lines in your changes are missing coverage. Please review.

Project coverage is 72.24%. Comparing base (1ff9649) to head (d07b542).

Files	Patch %	Lines
...g/opensearch/flowframework/model/WorkflowNode.java	18.18%	8 Missing and 1 partial ⚠️
.../org/opensearch/flowframework/util/ParseUtils.java	64.28%	3 Missing and 2 partials ⚠️
...ch/flowframework/workflow/CreateConnectorStep.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #558      +/-   ##
============================================
- Coverage     72.54%   72.24%   -0.31%     
+ Complexity      665      653      -12     
============================================
  Files            78       78              
  Lines          3416     3397      -19     
  Branches        271      269       -2     
============================================
- Hits           2478     2454      -24     
- Misses          822      825       +3     
- Partials        116      118       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbwiddis]

LGTM! A few suggestions.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/flow-framework/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/flow-framework/backport-2.x
# Create a new branch
git switch --create backport/backport-558-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b6d2092c35c5f94fd2ea644d375bfed345e0ede8
# Push it to GitHub
git push --set-upstream origin backport/backport-558-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/flow-framework/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-558-to-2.x.

[owaiskazi19]

@amitgalitz can you raise a manual backport PR?