[BUG] ClassCastException for Create Connector, Deploy Model, Model Group

[owaiskazi19]

What is the bug?

This issue is similar to the 3rd issue of #108 but for Create Connector and Register Model Group.

[2023-10-31T19:10:01,406][INFO ][o.r.Reflections          ] [ip-172-31-56-214] Reflections took 1 ms to scan 1 urls, producing 5 keys and 15 values 
[2023-10-31T19:10:01,412][INFO ][o.r.Reflections          ] [ip-172-31-56-214] Reflections took 4 ms to scan 1 urls, producing 11 keys and 49 values 
[2023-10-31T19:10:01,416][INFO ][o.r.Reflections          ] [ip-172-31-56-214] Reflections took 2 ms to scan 1 urls, producing 8 keys and 11 values 
[2023-10-31T19:10:01,449][INFO ][o.o.m.a.c.TransportCreateConnectorAction] [ip-172-31-56-214] connector created, indexing into the connector system index
[2023-10-31T19:10:01,452][INFO ][o.o.p.PluginsService     ] [ip-172-31-56-214] PluginService:onIndexModule index:[.plugins-ml-connector/Iib6OaZJQ4G1-MRjc0YYZw]
[2023-10-31T19:10:01,457][INFO ][o.o.c.m.MetadataCreateIndexService] [ip-172-31-56-214] [.plugins-ml-connector] creating index, cause [api], templates [], shards [1]/[1]
[2023-10-31T19:10:01,458][INFO ][o.o.c.r.a.AllocationService] [ip-172-31-56-214] updating number_of_replicas to [0] for indices [.plugins-ml-connector]
[2023-10-31T19:10:01,471][INFO ][o.o.p.PluginsService     ] [ip-172-31-56-214] PluginService:onIndexModule index:[.plugins-ml-connector/Iib6OaZJQ4G1-MRjc0YYZw]
[2023-10-31T19:10:01,499][INFO ][o.o.c.r.a.AllocationService] [ip-172-31-56-214] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-connector][0]]]).
[2023-10-31T19:10:01,511][INFO ][o.o.m.i.MLIndicesHandler ] [ip-172-31-56-214] create index:.plugins-ml-connector
[2023-10-31T19:10:01,530][INFO ][o.o.m.a.c.TransportCreateConnectorAction] [ip-172-31-56-214] Connector saved into index, result:CREATED, connector id: 0OAkh4sBoiuECYuvjQUp
[2023-10-31T19:10:01,531][ERROR][o.o.f.w.CreateConnectorStep] [ip-172-31-56-214] Failed to create connector
[2023-10-31T19:10:01,532][ERROR][o.o.f.t.ProvisionWorkflowTransportAction] [ip-172-31-56-214] Provisioning failed for workflow z-Akh4sBoiuECYuvOQX8 : org.opensearch.flowframework.exception.FlowFrameworkException: java.util.concurrent.ExecutionException: org.opensearch.flowframework.exception.FlowFrameworkException: class org.opensearch.ml.common.transport.connector.MLCreateConnectorResponse cannot be cast to class org.opensearch.ml.common.transport.connector.MLCreateConnectorResponse (org.opensearch.ml.common.transport.connector.MLCreateConnectorResponse is in unnamed module of loader java.net.FactoryURLClassLoader @5580d62f; org.opensearch.ml.common.transport.connector.MLCreateConnectorResponse is in unnamed module of loader java.net.FactoryURLClassLoader @10823d72)

The solution would be similar to make the changes on ml-commons side like: opensearch-project/ml-commons#1560 for Create Connector and Model Group.

How can one reproduce the bug?

1. Install ml-commons and flow-framework plugin to OS Cluster.
2. Hit the API

 curl -XPOST --insecure -u 'admin:admin' -H "Content-Type: application/json" 'http://localhost:9200/_plugins/_flow_framework/workflow' -d '{"name":"deploy-register-model","description":"test case","use_case":"SEMANTIC_SEARCH","version":{"template":"1.0.0","compatibility":["2.12.0","3.0.0"]},"workflows":{"provision":{"nodes":[{"id":"workflow_step_1","type":"create_connector","inputs":{"name":"OpenAI Chat Connector","description":"The connector to public OpenAI model service for GPT 3.5","version":"1","protocol":"http","parameters":{"endpoint":"api.openai.com","model":"gpt-3.5-turbo"},"credential":{"openAI_key":"..."},"actions":[{"action_type":"predict","method":"POST","url":"https://${parameters.endpoint}/v1/chat/completions"}]}},{"id":"workflow_step_2","type":"deploy_model","inputs":{"model_id":{}}}],"edges":[{"source":"workflow_step_1","dest":"workflow_step_2"}]}}}'
{"workflow_id":"z-Akh4sBoiuECYuvOQX8"}

3. Start provisioning

 curl -XPOST --insecure -u 'admin:admin' -H "Content-Type: application/json" 'http://localhost:9200/_plugins/_flow_framework/workflow/z-Akh4sBoiuECYuvOQX8/_provision'

What is the expected behavior?

A connector should be created and a model group should be registered.

What is your host/environment?

Ubuntu

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[joshpalis]

Additionally, there is a bug for the deploy model API of the MachineLearningNodeClient :

Provisioning failed for workflow LCifjIsBHB5sOr-SRvP8 : org.opensearch.flowframework.exception.FlowFrameworkException: java.util.concurrent.ExecutionException: org.opensearch.flowframework.exception.FlowFrameworkException: class org.opensearch.ml.common.transport.deploy.MLDeployModelResponse cannot be cast to class org.opensearch.ml.common.transport.deploy.MLDeployModelResponse (org.opensearch.ml.common.transport.deploy.MLDeployModelResponse is in unnamed module of loader java.net.FactoryURLClassLoader @17b6d426; org.opensearch.ml.common.transport.deploy.MLDeployModelResponse is in unnamed module of loader java.net.FactoryURLClassLoader @14b0e127)

I'll action this as well

[joshpalis]

PR opensearch-project/ml-commons#1580 has been raised to resolve this issue :

Testing full provision workflow locally with create connector, register model, deploy model steps :

Creating the workflow :

curl -i -XPOST "localhost:9200/_plugins/_flow_framework/workflow" -H "Content-Type:application/json" --data '{"name":"create-connector-register-deploy-model","description":"test case","use_case":"TEST_CASE","version":{"template":"1.0.0","compatibility":["2.12.0","3.0.0"]},"workflows":{"provision":{"nodes":[{"id":"workflow_step_1","type":"create_connector","user_inputs":{"name":"OpenAI Chat Connector","description":"The connector to public OpenAI model service for GPT 3.5","version":"1","protocol":"http","parameters":{"endpoint":"api.openai.com","model":"gpt-3.5-turbo"},"credential":{"openAI_key":"..."},"actions":[{"action_type":"predict","method":"POST","url":"https://${parameters.endpoint}/v1/chat/completions"}]}},{"id":"workflow_step_2","type":"register_model","previous_node_inputs":{"workflow_step_1":"connector_id"},"user_inputs":{"name":"openAI-gpt-3.5-turbo","function_name":"remote","description":"test model"}},{"id":"workflow_step_3","type":"deploy_model","previous_node_inputs":{"workflow_step_2":"model_id"}}],"edges":[{"source":"workflow_step_1","dest":"workflow_step_2"},{"source":"workflow_step_2","dest":"workflow_step_3"}]}}}'
HTTP/1.1 201 Created
X-OpenSearch-Version: OpenSearch/3.0.0-SNAPSHOT (opensearch)
content-type: application/json; charset=UTF-8
content-length: 38

{"workflow_id":"EdaujIsBaJDYTfB0hQJD"}

Provisioning workflow :

curl -i -XPOST "localhost:9200/_plugins/_flow_framework/workflow/EdaujIsBaJDYTfB0hQJD/_provision"
HTTP/1.1 200 OK
X-OpenSearch-Version: OpenSearch/3.0.0-SNAPSHOT (opensearch)
content-type: application/json; charset=UTF-8
content-length: 38

{"workflow_id":"EdaujIsBaJDYTfB0hQJD"}

OpenSearch logs successful completion of all workflow steps : Provisioning completed successuflly for workflow EdaujIsBaJDYTfB0hQJD

[2023-11-01T20:58:49,537][INFO ][o.o.f.i.FlowFrameworkIndicesHandler] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] create index:.plugins-ai-global-context
[2023-11-01T20:58:49,583][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-workflow-state/5Y85uH9nRsm2fOIILM98Jw]
[2023-11-01T20:58:49,596][INFO ][o.o.c.m.MetadataCreateIndexService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] [.plugins-workflow-state] creating index, cause [api], templates [], shards [1]/[1]
[2023-11-01T20:58:49,598][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] updating number_of_replicas to [0] for indices [.plugins-workflow-state]
[2023-11-01T20:58:49,620][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-workflow-state/5Y85uH9nRsm2fOIILM98Jw]
[2023-11-01T20:58:49,662][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-workflow-state][0]]]).
[2023-11-01T20:58:49,680][INFO ][o.o.f.i.FlowFrameworkIndicesHandler] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] create index:.plugins-workflow-state
[2023-11-01T20:58:49,693][INFO ][o.o.f.t.CreateWorkflowTransportAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] create state workflow doc
[2023-11-01T20:59:07,901][INFO ][o.o.f.t.ProvisionWorkflowTransportAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Queueing process [workflow_step_1]. Can start immediately!
[2023-11-01T20:59:07,903][INFO ][o.o.f.t.ProvisionWorkflowTransportAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Queueing process [workflow_step_2]. Must wait for [workflow_step_1] to complete first.
[2023-11-01T20:59:07,903][INFO ][o.o.f.w.ProcessNode      ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Starting workflow_step_1.
[2023-11-01T20:59:07,903][INFO ][o.o.f.t.ProvisionWorkflowTransportAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Queueing process [workflow_step_3]. Must wait for [workflow_step_2] to complete first.
[2023-11-01T20:59:07,921][INFO ][o.r.Reflections          ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Reflections took 8 ms to scan 1 urls, producing 6 keys and 34 values 
[2023-11-01T20:59:07,931][INFO ][o.r.Reflections          ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Reflections took 7 ms to scan 1 urls, producing 8 keys and 38 values 
[2023-11-01T20:59:07,934][INFO ][o.o.f.t.ProvisionWorkflowTransportAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] updated workflow EdaujIsBaJDYTfB0hQJD state to PROVISIONING
[2023-11-01T20:59:07,936][INFO ][o.r.Reflections          ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Reflections took 2 ms to scan 1 urls, producing 5 keys and 14 values 
[2023-11-01T20:59:07,941][INFO ][o.r.Reflections          ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Reflections took 2 ms to scan 1 urls, producing 3 keys and 8 values 
[2023-11-01T20:59:07,946][INFO ][o.r.Reflections          ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Reflections took 3 ms to scan 1 urls, producing 5 keys and 15 values 
[2023-11-01T20:59:07,957][INFO ][o.r.Reflections          ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Reflections took 8 ms to scan 1 urls, producing 11 keys and 49 values 
[2023-11-01T20:59:07,963][INFO ][o.r.Reflections          ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Reflections took 3 ms to scan 1 urls, producing 8 keys and 11 values 
[2023-11-01T20:59:08,004][INFO ][o.o.m.a.c.TransportCreateConnectorAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] connector created, indexing into the connector system index
[2023-11-01T20:59:08,008][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-ml-connector/mkvQpiBkRHicZA9y0Lj8YQ]
[2023-11-01T20:59:08,019][INFO ][o.o.c.m.MetadataCreateIndexService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] [.plugins-ml-connector] creating index, cause [api], templates [], shards [1]/[1]
[2023-11-01T20:59:08,020][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] updating number_of_replicas to [0] for indices [.plugins-ml-connector]
[2023-11-01T20:59:08,041][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-ml-connector/mkvQpiBkRHicZA9y0Lj8YQ]
[2023-11-01T20:59:08,089][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-connector][0]]]).
[2023-11-01T20:59:08,107][INFO ][o.o.m.i.MLIndicesHandler ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] create index:.plugins-ml-connector
[2023-11-01T20:59:08,130][INFO ][o.o.m.a.c.TransportCreateConnectorAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Connector saved into index, result:CREATED, connector id: EtaujIsBaJDYTfB0zQLN
[2023-11-01T20:59:08,131][INFO ][o.o.f.w.CreateConnectorStep] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Created connector successfully
[2023-11-01T20:59:08,131][INFO ][o.o.f.w.ProcessNode      ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Finished workflow_step_1.
[2023-11-01T20:59:08,131][INFO ][o.o.f.w.ProcessNode      ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Starting workflow_step_2.
[2023-11-01T20:59:08,151][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-ml-model-group/mVNad_2QSLqHZpVRbVXw-w]
[2023-11-01T20:59:08,156][INFO ][o.o.c.m.MetadataCreateIndexService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] [.plugins-ml-model-group] creating index, cause [api], templates [], shards [1]/[1]
[2023-11-01T20:59:08,157][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] updating number_of_replicas to [0] for indices [.plugins-ml-model-group]
[2023-11-01T20:59:08,175][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-ml-model-group/mVNad_2QSLqHZpVRbVXw-w]
[2023-11-01T20:59:08,221][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-model-group][0]]]).
[2023-11-01T20:59:08,238][INFO ][o.o.m.i.MLIndicesHandler ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] create index:.plugins-ml-model-group
[2023-11-01T20:59:08,255][INFO ][stdout                   ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] registering the model
[2023-11-01T20:59:08,259][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-ml-task/vUP_B5jWR8yZAPlX2kNELA]
[2023-11-01T20:59:08,265][INFO ][o.o.c.m.MetadataCreateIndexService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] [.plugins-ml-task] creating index, cause [api], templates [], shards [1]/[1]
[2023-11-01T20:59:08,266][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] updating number_of_replicas to [0] for indices [.plugins-ml-task]
[2023-11-01T20:59:08,283][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-ml-task/vUP_B5jWR8yZAPlX2kNELA]
[2023-11-01T20:59:08,329][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-task][0]]]).
[2023-11-01T20:59:08,346][INFO ][o.o.m.i.MLIndicesHandler ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] create index:.plugins-ml-task
[2023-11-01T20:59:08,384][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-ml-model/AxPVDVOfT12WB8srj-eePQ]
[2023-11-01T20:59:08,394][INFO ][o.o.c.m.MetadataCreateIndexService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] [.plugins-ml-model] creating index, cause [api], templates [], shards [1]/[1]
[2023-11-01T20:59:08,395][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] updating number_of_replicas to [0] for indices [.plugins-ml-model]
[2023-11-01T20:59:08,412][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-ml-model/AxPVDVOfT12WB8srj-eePQ]
[2023-11-01T20:59:08,452][INFO ][o.o.c.r.a.AllocationService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-model][0]]]).
[2023-11-01T20:59:08,468][INFO ][o.o.m.i.MLIndicesHandler ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] create index:.plugins-ml-model
[2023-11-01T20:59:08,480][INFO ][o.o.p.PluginsService     ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] PluginService:onIndexModule index:[.plugins-ml-model/AxPVDVOfT12WB8srj-eePQ]
[2023-11-01T20:59:08,488][INFO ][o.o.c.m.MetadataMappingService] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] [.plugins-ml-model/AxPVDVOfT12WB8srj-eePQ] update_mapping [_doc]
[2023-11-01T20:59:08,527][INFO ][o.o.m.m.MLModelManager   ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] create new model meta doc FdaujIsBaJDYTfB0zwI2 for upload task FNaujIsBaJDYTfB0zgK8
[2023-11-01T20:59:08,530][INFO ][o.o.f.w.RegisterModelStep] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Model registration successful
[2023-11-01T20:59:08,531][INFO ][o.o.f.w.ProcessNode      ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Finished workflow_step_2.
[2023-11-01T20:59:08,531][INFO ][o.o.f.w.ProcessNode      ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Starting workflow_step_3.
[2023-11-01T20:59:08,541][INFO ][o.o.m.a.d.TransportDeployModelAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Will deploy model on these nodes: KjtqJSkTSISCXGLthC9oAQ
[2023-11-01T20:59:08,581][INFO ][o.o.m.m.MLModelManager   ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Set connector EtaujIsBaJDYTfB0zQLN for the model: FdaujIsBaJDYTfB0zwI2
[2023-11-01T20:59:08,582][INFO ][o.o.f.w.DeployModelStep  ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Model deployment state COMPLETED
[2023-11-01T20:59:08,583][INFO ][o.o.f.w.ProcessNode      ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Finished workflow_step_3.
[2023-11-01T20:59:08,583][INFO ][o.o.f.t.ProvisionWorkflowTransportAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Provisioning completed successuflly for workflow EdaujIsBaJDYTfB0hQJD
[2023-11-01T20:59:08,619][INFO ][o.o.m.a.f.TransportForwardAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] deploy model done with state: DEPLOYED, model id: FdaujIsBaJDYTfB0zwI2
[2023-11-01T20:59:08,620][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] deploy model task done FtaujIsBaJDYTfB0zwKB
[2023-11-01T20:59:08,621][INFO ][o.o.m.m.MLModelManager   ] [dev-dsk-jpalis-2c-27c8aa11.us-west-2.amazon.com] Completed setting connector EtaujIsBaJDYTfB0zQLN in the model FdaujIsBaJDYTfB0zwI2