[BUG] Deprovision doesn't remove the error field if it exists

[amitgalitz]

What is the bug?

If provisioning has a failure on one of the steps we write to the error field in the state index. However when we deprovision that same workflow we don't remove the error message from the document in the workflow state index.

How can one reproduce the bug?

1. Provision with an expected failure (easy way is to provision a pipeline with a processor that isn't in your cluster)
   2.Deprovision the same workflow:

resulting state index:

            {
                "_index": ".plugins-flow-framework-state",
                "_id": "QljrPY4BJDcxHyOJPUNd",
                "_score": 1.0,
                "_source": {
                    "workflow_id": "QljrPY4BJDcxHyOJPUNd",
                    "state": "NOT_STARTED",
                    "provisioning_progress": "NOT_STARTED",
                    "provision_start_time": 1710435961112,
                    "resources_created": [],
                    "provision_end_time": 1710435828966,
                    "error": "org.opensearch.flowframework.exception.WorkflowStepException during step create_ingest_pipeline, restStatus: BAD_REQUEST"
                }
            }
            

What is the expected behavior?

error field shouldn't be there anymore

Potential solution

Some potential solution

1. We can either simply place the error field as an empty string, however this isn't prefered because if the error field wasn't there to begin with now we have the field with just "error": "".
2. You can't directly remove the field through update, this has to be done through a script as far as I know. You also can't provide both a contentMap(doc) and a script in the same update request. This means we either have to just update the logic here https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/transport/DeprovisionWorkflowTransportAction.java#L230 to update with a script that removes the error field if it exists (e.g "if (ctx._source.contains(error)) { ctx._source.remove('error)}") and updates the other fields accordingly or we have to make two update operations, one with the current way we do it with a new doc and another with the painless script

[dbwiddis]

It's not a bug, it's a feature! 😁

Seriously, though, this is very interesting. I'm not overly concerned a provisioning error hangs around after deprovisioning the failure (although should we re-populate it with deprovisioning failures, that I don't think we do?). More importantly if I provision a second time does the error remain even if provisioning was successful?

[dbwiddis]

While implementing #631 I couldn't use the field-by-field update or script, so I added some helper methods to WorkflowState that would enable this:

• Create a builder using the existing state
• update the fields as necessary (e.g., set error to null)
• just index the new document to overwrite the old one, instead of using update.

See https://github.com/opensearch-project/flow-framework/pull/631/files#diff-d49816ce76f874e242a545dd3de0515c2435ec7a6a315faf83ae28e4069f08e0, some of those changes should probably be kept.