[BUG] Validation of installed plugins blocks forever

[dbwiddis]

What is the bug?

The check for installed plugins may block indefinitely on some clusters.

flow-framework/src/main/java/org/opensearch/flowframework/workflow/WorkflowProcessSorter.java

Line 180 in dd32d55

List<String> installedPlugins = installedPluginsFuture.get();

How can one reproduce the bug?

Create a 2-node cluster on macOS using main (3.0.0-SNAPSHOT) branches.

What is the expected behavior?

Validation fails after some timeout with 409 error identifying the cause.

Do you have any additional context?

"opensearch[node-1][transport_worker][T#4]" #47 [36355] daemon prio=5 os_prio=31 cpu=7494.81ms elapsed=12333.24s tid=0x000000012e55f200 nid=36355 waiting on condition  [0x00000001796c5000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x00000007e91cb998> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:221)
	at java.util.concurrent.CompletableFuture$Signaller.block([email protected]/CompletableFuture.java:1864)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock([email protected]/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3725)
	at java.util.concurrent.CompletableFuture.waitingGet([email protected]/CompletableFuture.java:1898)
	at java.util.concurrent.CompletableFuture.get([email protected]/CompletableFuture.java:2072)
	at org.opensearch.flowframework.workflow.WorkflowProcessSorter.validatePluginsInstalled(WorkflowProcessSorter.java:180)
	at org.opensearch.flowframework.workflow.WorkflowProcessSorter.validate(WorkflowProcessSorter.java:145)
	at org.opensearch.flowframework.transport.CreateWorkflowTransportAction.validateWorkflows(CreateWorkflowTransportAction.java:266)
	at org.opensearch.flowframework.transport.CreateWorkflowTransportAction.doExecute(CreateWorkflowTransportAction.java:102)
	at org.opensearch.flowframework.transport.CreateWorkflowTransportAction.doExecute(CreateWorkflowTransportAction.java:50)
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)