[META] End to end template API user experience

[dbwiddis]

The APIs to save and provision a workflow as part of #49 need to be fully integrated with existing and needed validation checks and exception handling. The goal is to provide fast-failure validation information to a user giving them enough information to correct an invalid template, and also provide appropriate feedback to a user when a workflow fails, identifying enough detail to correct the issue.

The following steps need to be completed:

P0 Tasks

• Update existing workflow parsing code to replace IllegalArgumentException with a FlowFrameworkException including an appropriate RestStatus. #656
• Update existing workflow process execution code to handle the cause of exceptionally completed futures (TimeoutException or other failures) with a FlowFrameworkException including an appropriate RestStatus.
• Update the RestResponse for the API method(s) to handle these exceptions, and build a response to the user containing an appropriately detailed error message.
• Add input/output data validation checks to the workflow parsing. [FEATURE] Validate a process flow graph #18 and [FEATURE] Define required input and provided output keys #67. These may involve updating the workflow step factory.
• [FEATURE] Remove user_inputs field from template and set values directly for each workflow #105
• [FEATURE] RegisterModelStep / DeployModelStep Improvements #108
• [BUG] TypeCast issue for Create Connector #123
• [BUG] ClassCastException for Create Connector, Deploy Model, Model Group  #131
• [FEATURE] Separate RegisterModelStep into RegsiterRemoteModelStep and RegisterLocalModelStep #149
• Handle optional params for Register Model, Deploy Model and Create Connector similar to done in Model Group Step
• [FEATURE] Create Status API #22
• [FEATURE] Add transport request retry capability for async workflow steps #158
• [FEATURE] Handle Credentials in Global Context #75
• Combine RegisterLocalModelStep and GetMLTaskStep (Covered by PR Combined register local model and get ml task #220)
• [FEATURE] WorkflowStep Execution Improvements #210
• Integrate tests of the above features with Integration Testing. Refactor tests as necessary to either mock or make use of the OpenSearch test framework's clients
• [FEATURE] Get Workflow API #170
• [FEATURE] Provide an API to search for a resource by workflow step id #279
• [Feature/agent_framework] Adding installed plugins validation #290

P1 Tasks

• [FEATURE] Add a dry-run parameter to the RestCreateWorkflowAction to validate the template workflow graph  #99
• [FEATURE] CreateIndexStep improvements #103
• [FEATURE] CreateIngestPipelineStep Improvements #104

[joshpalis]

Refactor parsing steps as necessary to generate failures as early as possible in the process. Consider adding a "dryrun" parameter to the template saving APIs for the user to perform validation without making any changes.

This is a good idea, perhaps we can run the template through the workflow process sorter to validate the graph and not execute it

[dbwiddis]

Perhaps we can run the template through the workflow process sorter to validate the graph and not execute it

Yep, there's a private method we can probably make public to more easily enable that.

[owaiskazi19]

Regarding the 4th point, trying to understand more for validation. Should we have a dry run param or validation should be done every time there's a call to template API? Be it to just to save the template or start the provisioning.

[joshpalis]

Regarding the 4th point, trying to understand more for validation. Should we have a dry run param or validation should be done every time there's a call to template API? Be it to just to save the template or start the provisioning.|

If we add validation by default as a prerequisite to saving a WIP template, that could be a frustrating experience for a builder. In that regard, I think it should be a separate capability and having an optional dry-run param for the RestCreateWorkflowAction to validate prior to saving makes sense.

Having the dry-run param for the RestProvisionWorkflowAction wouldn't make sense here, since validation is done already when we topologically sort the workflow.

[dbwiddis]

If we add validation by default as a prerequisite to saving a WIP template, that could be a frustrating experience for a builder. In that regard, I think it should be a separate capability and having an optional dry-run param for the RestCreateWorkflowAction to validate prior to saving makes sense.

I think validation goes more with the "execution" portion, but we should be able to do the validation before saving. Which means we sort of need a "save-and-execute" single API call.

Which brings back up the possibility of combining APIs.

We have

POST _plugins/_flow_framework/workflow
{
    // use case template
}

followed by

POST _plugins/_flow_framework/workflow/<workflow_id>/_provision
{}

This works well for a "save WIP and finally execute when done" but in the case we have a completed/debugged template that we always want to execute, we could do that all-in-one with:

POST _plugins/_flow_framework/workflow/_provision
{
    // use case template
}

Now, throw a validation param in there and we have a one-liner just like we want:

POST _plugins/_flow_framework/workflow/_provision?dryrun=true
{
    // use case template
}

(or similar)

[joshpalis]

Closing as completed since the remaining P1 items have separate issues tracking them