Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for reroute operation during node-left #16468

Merged
merged 1 commit into from
Oct 28, 2024

Conversation

rajiv-kv
Copy link
Contributor

@rajiv-kv rajiv-kv commented Oct 24, 2024

Description

[Describe what this change achieves]

During node-left task execution, the shards on the leaving node are failed and reroute operation is triggered. The input RoutingAllocation to reroute has updated ShardRouting information. As part of reroute, the existing recoveries are looked for a better match using the ShardRouting cached in ShardBatch cache. Since this can contain the stale ShardRoutring assigned to the node leaving the cluster, it causes NPE when the node is referenced during shard allocation decisions.

Repro for stale batch cache entries as compared to RoutingAllocation


[2024-10-24T09:15:54,708][INFO ][o.o.c.c.RareClusterStateIT] [node_t0] removed the node uwpG13DpRq-ZvWOUbgpivA
[2024-10-24T09:15:54,708][INFO ][o.o.c.c.RareClusterStateIT] [node_t0] shard [index][2], node[IpyffjDySMu71qtMsnaBsg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=3GndaTH0QtKLVzCMMg4R_Q], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:15:54.340Z], delayed=false, allocation_status[no_attempt]]
[2024-10-24T09:16:52,868][INFO ][o.o.g.S.InternalReplicaBatchShardAllocator] [node_t0] [index][2], node[IpyffjDySMu71qtMsnaBsg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=3GndaTH0QtKLVzCMMg4R_Q], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:15:54.340Z], delayed=false, allocation_status[no_attempt]]: no active primary shard found or allocated, letting actual allocation figure it out

node uwpG13DpRq-ZvWOUbgpivA is removed from cluster and the ShardAllocator tries to reference the PrimaryShard [index][2] which was previously allocated on the node. The batch cache has Replica shard for [index][2] still in INITIALIZING state although it has been marked as UNASSIGNED in RoutingAllocation.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

❌ Gradle check result for 8a64664: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 96c0766: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@rajiv-kv
Copy link
Contributor Author

[2024-10-24T04:35:25,753][INFO ][o.o.t.OpenSearchTestClusterRule] [testDisassociateNodesWhileShardInit] [RareClusterStateIT#testDisassociateNodesWhileShardInit]: setting up test
[2024-10-24T04:35:25,798][INFO ][o.o.t.InternalTestCluster] [testDisassociateNodesWhileShardInit] Setup InternalTestCluster [TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster] with seed [EF5B56995926D37D] using [0] dedicated cluster-managers, [0] (data) nodes and [0] coord only nodes (min_cluster_manager_nodes are [auto-managed])
[2024-10-24T04:35:25,925][INFO ][o.o.t.OpenSearchTestClusterRule] [testDisassociateNodesWhileShardInit] [RareClusterStateIT#testDisassociateNodesWhileShardInit]: all set up test
[2024-10-24T04:35:25,925][INFO ][o.o.c.c.RareClusterStateIT] [testDisassociateNodesWhileShardInit] before test
[2024-10-24T04:35:26,036][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] version[3.0.0], pid[83902], build[unknown/unknown/unknown], OS[Mac OS X/13.6.8/aarch64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.21/11.0.21+9]
[2024-10-24T04:35:26,037][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] JVM home [/Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home]
[2024-10-24T04:35:26,040][DEPRECATION][o.o.d.n.Node             ] [testDisassociateNodesWhileShardInit] no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
[2024-10-24T04:35:26,040][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] JVM arguments [-ea, -Djava.locale.providers=SPI,CLDR, -Didea.test.cyclic.buffer.size=1048576, -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=64822:/Applications/IntelliJ IDEA.app/Contents/bin, -Dfile.encoding=UTF-8]
[2024-10-24T04:35:26,050][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] no modules loaded
[2024-10-24T04:35:26,051][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin]
[2024-10-24T04:35:26,051][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-10-24T04:35:26,051][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-10-24T04:35:26,051][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-10-24T04:35:26,051][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-10-24T04:35:26,051][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-10-24T04:35:26,051][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-10-24T04:35:26,051][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-10-24T04:35:26,051][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-10-24T04:35:26,067][INFO ][o.o.e.NodeEnvironment    ] [testDisassociateNodesWhileShardInit] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk3s5)]], net usable_space [233.7gb], net total_space [460.4gb], types [apfs]
[2024-10-24T04:35:26,068][INFO ][o.o.e.NodeEnvironment    ] [testDisassociateNodesWhileShardInit] heap size [8gb], compressed ordinary object pointers [true]
[2024-10-24T04:35:26,103][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] node name [node_t0], node ID [wno7Y3oFSmW10pPq5Fkn8g], cluster name [TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster], roles [cluster_manager]
[2024-10-24T04:35:26,129][INFO ][o.o.e.ExtensionsManager  ] [testDisassociateNodesWhileShardInit] ExtensionsManager initialized
[2024-10-24T04:35:27,465][INFO ][o.o.d.DiscoveryModule    ] [testDisassociateNodesWhileShardInit] using discovery type [zen] and seed hosts providers [settings, file]
[2024-10-24T04:35:27,805][WARN ][o.o.g.DanglingIndicesState] [testDisassociateNodesWhileShardInit] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-10-24T04:35:27,957][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] initialized
[2024-10-24T04:35:27,959][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#1]]] starting ...
[2024-10-24T04:35:27,989][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#1]]] publish_address {127.0.0.1:64832}, bound_addresses {[::1]:64831}, {127.0.0.1:64832}
[2024-10-24T04:35:28,157][INFO ][o.o.c.c.Coordinator      ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#1]]] setting initial configuration to VotingConfiguration{wno7Y3oFSmW10pPq5Fkn8g}
[2024-10-24T04:35:28,343][INFO ][o.o.c.s.MasterService    ] [node_t0] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper and count: 3, term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}]}
[2024-10-24T04:35:28,382][INFO ][o.o.c.c.CoordinationState] [node_t0] cluster UUID set to [fHv2u1xFRsqPJR57v4bS8w]
[2024-10-24T04:35:28,410][INFO ][o.o.c.s.ClusterApplierService] [node_t0] cluster-manager node changed {previous [], current [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2024-10-24T04:35:28,421][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#1]]] started
[2024-10-24T04:35:28,426][INFO ][o.o.d.PeerFinder         ] [node_t0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}]
[2024-10-24T04:35:28,431][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] version[3.0.0], pid[83902], build[unknown/unknown/unknown], OS[Mac OS X/13.6.8/aarch64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.21/11.0.21+9]
[2024-10-24T04:35:28,431][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] JVM home [/Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home]
[2024-10-24T04:35:28,431][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] JVM arguments [-ea, -Djava.locale.providers=SPI,CLDR, -Didea.test.cyclic.buffer.size=1048576, -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=64822:/Applications/IntelliJ IDEA.app/Contents/bin, -Dfile.encoding=UTF-8]
[2024-10-24T04:35:28,431][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] no modules loaded
[2024-10-24T04:35:28,431][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin]
[2024-10-24T04:35:28,431][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-10-24T04:35:28,431][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-10-24T04:35:28,431][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-10-24T04:35:28,431][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-10-24T04:35:28,432][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-10-24T04:35:28,432][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-10-24T04:35:28,432][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-10-24T04:35:28,432][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-10-24T04:35:28,434][INFO ][o.o.e.NodeEnvironment    ] [testDisassociateNodesWhileShardInit] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk3s5)]], net usable_space [233.7gb], net total_space [460.4gb], types [apfs]
[2024-10-24T04:35:28,434][INFO ][o.o.e.NodeEnvironment    ] [testDisassociateNodesWhileShardInit] heap size [8gb], compressed ordinary object pointers [true]
[2024-10-24T04:35:28,438][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] node name [node_t1], node ID [sIJZ0V2STR2kUV2TTe6nTg], cluster name [TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster], roles [data]
[2024-10-24T04:35:28,440][INFO ][o.o.e.ExtensionsManager  ] [testDisassociateNodesWhileShardInit] ExtensionsManager initialized
[2024-10-24T04:35:28,442][INFO ][o.o.g.GatewayService     ] [node_t0] recovered [0] indices into cluster_state
[2024-10-24T04:35:28,452][INFO ][o.o.d.DiscoveryModule    ] [testDisassociateNodesWhileShardInit] using discovery type [zen] and seed hosts providers [settings, file]
[2024-10-24T04:35:28,466][WARN ][o.o.g.DanglingIndicesState] [testDisassociateNodesWhileShardInit] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-10-24T04:35:28,483][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] initialized
[2024-10-24T04:35:28,485][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#2]]] starting ...
[2024-10-24T04:35:28,491][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#2]]] publish_address {127.0.0.1:64835}, bound_addresses {[::1]:64834}, {127.0.0.1:64835}
[2024-10-24T04:35:28,580][INFO ][o.o.c.s.MasterService    ] [node_t0] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper and count: 1, term: 1, version: 3, delta: added {{node_t1}{sIJZ0V2STR2kUV2TTe6nTg}{7aCEwCzFRx6MIPU33XT2jg}{127.0.0.1}{127.0.0.1:64835}{d}{shard_indexing_pressure_enabled=true}}
[2024-10-24T04:35:28,598][INFO ][o.o.c.s.ClusterApplierService] [node_t1] cluster-manager node changed {previous [], current [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}]}, added {{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}}, term: 1, version: 3, reason: ApplyCommitRequest{term=1, version=3, sourceNode={node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}}
[2024-10-24T04:35:28,599][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#2]]] started
[2024-10-24T04:35:28,600][INFO ][o.o.c.s.ClusterApplierService] [node_t0] added {{node_t1}{sIJZ0V2STR2kUV2TTe6nTg}{7aCEwCzFRx6MIPU33XT2jg}{127.0.0.1}{127.0.0.1:64835}{d}{shard_indexing_pressure_enabled=true}}, term: 1, version: 3, reason: Publication{term=1, version=3}
[2024-10-24T04:35:28,601][INFO ][o.o.d.PeerFinder         ] [node_t1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t1}{sIJZ0V2STR2kUV2TTe6nTg}{7aCEwCzFRx6MIPU33XT2jg}{127.0.0.1}{127.0.0.1:64835}{d}{shard_indexing_pressure_enabled=true}]
[2024-10-24T04:35:28,605][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] version[3.0.0], pid[83902], build[unknown/unknown/unknown], OS[Mac OS X/13.6.8/aarch64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.21/11.0.21+9]
[2024-10-24T04:35:28,605][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] JVM home [/Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home]
[2024-10-24T04:35:28,605][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] JVM arguments [-ea, -Djava.locale.providers=SPI,CLDR, -Didea.test.cyclic.buffer.size=1048576, -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=64822:/Applications/IntelliJ IDEA.app/Contents/bin, -Dfile.encoding=UTF-8]
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] no modules loaded
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin]
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-10-24T04:35:28,605][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-10-24T04:35:28,608][INFO ][o.o.e.NodeEnvironment    ] [testDisassociateNodesWhileShardInit] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk3s5)]], net usable_space [233.7gb], net total_space [460.4gb], types [apfs]
[2024-10-24T04:35:28,608][INFO ][o.o.e.NodeEnvironment    ] [testDisassociateNodesWhileShardInit] heap size [8gb], compressed ordinary object pointers [true]
[2024-10-24T04:35:28,612][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] node name [node_t2], node ID [bpTysqApSV6NKNVTVxuCNA], cluster name [TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster], roles [data]
[2024-10-24T04:35:28,613][INFO ][o.o.e.ExtensionsManager  ] [testDisassociateNodesWhileShardInit] ExtensionsManager initialized
[2024-10-24T04:35:28,638][INFO ][o.o.d.DiscoveryModule    ] [testDisassociateNodesWhileShardInit] using discovery type [zen] and seed hosts providers [settings, file]
[2024-10-24T04:35:28,667][WARN ][o.o.g.DanglingIndicesState] [testDisassociateNodesWhileShardInit] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-10-24T04:35:28,675][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] initialized
[2024-10-24T04:35:28,677][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#3]]] starting ...
[2024-10-24T04:35:28,682][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#3]]] publish_address {127.0.0.1:64844}, bound_addresses {[::1]:64843}, {127.0.0.1:64844}
[2024-10-24T04:35:28,720][INFO ][o.o.c.s.MasterService    ] [node_t0] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper and count: 1, term: 1, version: 4, delta: added {{node_t2}{bpTysqApSV6NKNVTVxuCNA}{j2G6tAb6QdeUUdroD2DLoQ}{127.0.0.1}{127.0.0.1:64844}{d}{shard_indexing_pressure_enabled=true}}
[2024-10-24T04:35:28,727][INFO ][o.o.c.s.ClusterApplierService] [node_t1] added {{node_t2}{bpTysqApSV6NKNVTVxuCNA}{j2G6tAb6QdeUUdroD2DLoQ}{127.0.0.1}{127.0.0.1:64844}{d}{shard_indexing_pressure_enabled=true}}, term: 1, version: 4, reason: ApplyCommitRequest{term=1, version=4, sourceNode={node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}}
[2024-10-24T04:35:28,727][INFO ][o.o.c.s.ClusterApplierService] [node_t2] cluster-manager node changed {previous [], current [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}]}, added {{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true},{node_t1}{sIJZ0V2STR2kUV2TTe6nTg}{7aCEwCzFRx6MIPU33XT2jg}{127.0.0.1}{127.0.0.1:64835}{d}{shard_indexing_pressure_enabled=true}}, term: 1, version: 4, reason: ApplyCommitRequest{term=1, version=4, sourceNode={node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}}
[2024-10-24T04:35:28,731][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#3]]] started
[2024-10-24T04:35:28,731][INFO ][o.o.c.s.ClusterApplierService] [node_t0] added {{node_t2}{bpTysqApSV6NKNVTVxuCNA}{j2G6tAb6QdeUUdroD2DLoQ}{127.0.0.1}{127.0.0.1:64844}{d}{shard_indexing_pressure_enabled=true}}, term: 1, version: 4, reason: Publication{term=1, version=4}
[2024-10-24T04:35:28,733][INFO ][o.o.d.PeerFinder         ] [node_t2] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t2}{bpTysqApSV6NKNVTVxuCNA}{j2G6tAb6QdeUUdroD2DLoQ}{127.0.0.1}{127.0.0.1:64844}{d}{shard_indexing_pressure_enabled=true}]
[2024-10-24T04:35:28,736][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] version[3.0.0], pid[83902], build[unknown/unknown/unknown], OS[Mac OS X/13.6.8/aarch64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.21/11.0.21+9]
[2024-10-24T04:35:28,736][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] JVM home [/Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home]
[2024-10-24T04:35:28,736][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] JVM arguments [-ea, -Djava.locale.providers=SPI,CLDR, -Didea.test.cyclic.buffer.size=1048576, -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=64822:/Applications/IntelliJ IDEA.app/Contents/bin, -Dfile.encoding=UTF-8]
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] no modules loaded
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.cluster.service.applicationtemplates.TestSystemTemplatesRepositoryPlugin]
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.script.MockScriptService$TestPlugin]
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.MockHttpTransport$TestPlugin]
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$AssertActionNamePlugin]
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.OpenSearchIntegTestCase$TestSeedPlugin]
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.TestGeoShapeFieldMapperPlugin]
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.telemetry.MockTelemetryPlugin]
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.test.transport.MockTransportService$TestPlugin]
[2024-10-24T04:35:28,737][INFO ][o.o.p.PluginsService     ] [testDisassociateNodesWhileShardInit] loaded plugin [org.opensearch.transport.nio.MockNioTransportPlugin]
[2024-10-24T04:35:28,740][INFO ][o.o.e.NodeEnvironment    ] [testDisassociateNodesWhileShardInit] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk3s5)]], net usable_space [233.7gb], net total_space [460.4gb], types [apfs]
[2024-10-24T04:35:28,740][INFO ][o.o.e.NodeEnvironment    ] [testDisassociateNodesWhileShardInit] heap size [8gb], compressed ordinary object pointers [true]
[2024-10-24T04:35:28,745][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] node name [node_t3], node ID [15gcWzhAS3e7yp-88UHxfQ], cluster name [TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster], roles [data]
[2024-10-24T04:35:28,746][INFO ][o.o.e.ExtensionsManager  ] [testDisassociateNodesWhileShardInit] ExtensionsManager initialized
[2024-10-24T04:35:28,758][INFO ][o.o.d.DiscoveryModule    ] [testDisassociateNodesWhileShardInit] using discovery type [zen] and seed hosts providers [settings, file]
[2024-10-24T04:35:28,768][WARN ][o.o.g.DanglingIndicesState] [testDisassociateNodesWhileShardInit] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-10-24T04:35:28,775][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] initialized
[2024-10-24T04:35:28,776][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#4]]] starting ...
[2024-10-24T04:35:28,780][INFO ][o.o.t.TransportService   ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#4]]] publish_address {127.0.0.1:64859}, bound_addresses {[::1]:64858}, {127.0.0.1:64859}
[2024-10-24T04:35:28,821][INFO ][o.o.c.s.MasterService    ] [node_t0] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper and count: 1, term: 1, version: 5, delta: added {{node_t3}{15gcWzhAS3e7yp-88UHxfQ}{pCVUdUvES--Usb2aUyosJA}{127.0.0.1}{127.0.0.1:64859}{d}{shard_indexing_pressure_enabled=true}}
[2024-10-24T04:35:28,829][INFO ][o.o.c.s.ClusterApplierService] [node_t2] added {{node_t3}{15gcWzhAS3e7yp-88UHxfQ}{pCVUdUvES--Usb2aUyosJA}{127.0.0.1}{127.0.0.1:64859}{d}{shard_indexing_pressure_enabled=true}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}}
[2024-10-24T04:35:28,829][INFO ][o.o.c.s.ClusterApplierService] [node_t3] cluster-manager node changed {previous [], current [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}]}, added {{node_t2}{bpTysqApSV6NKNVTVxuCNA}{j2G6tAb6QdeUUdroD2DLoQ}{127.0.0.1}{127.0.0.1:64844}{d}{shard_indexing_pressure_enabled=true},{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true},{node_t1}{sIJZ0V2STR2kUV2TTe6nTg}{7aCEwCzFRx6MIPU33XT2jg}{127.0.0.1}{127.0.0.1:64835}{d}{shard_indexing_pressure_enabled=true}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}}
[2024-10-24T04:35:28,829][INFO ][o.o.c.s.ClusterApplierService] [node_t1] added {{node_t3}{15gcWzhAS3e7yp-88UHxfQ}{pCVUdUvES--Usb2aUyosJA}{127.0.0.1}{127.0.0.1:64859}{d}{shard_indexing_pressure_enabled=true}}, term: 1, version: 5, reason: ApplyCommitRequest{term=1, version=5, sourceNode={node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}}
[2024-10-24T04:35:28,835][INFO ][o.o.n.Node               ] [[test_TEST-TEST_WORKER_VM=[--not-gradle--]-CLUSTER_SEED=[-1199269659138403459]-HASH=[22B449AB37208]-cluster[T#4]]] started
[2024-10-24T04:35:28,835][INFO ][o.o.c.s.ClusterApplierService] [node_t0] added {{node_t3}{15gcWzhAS3e7yp-88UHxfQ}{pCVUdUvES--Usb2aUyosJA}{127.0.0.1}{127.0.0.1:64859}{d}{shard_indexing_pressure_enabled=true}}, term: 1, version: 5, reason: Publication{term=1, version=5}
[2024-10-24T04:35:28,837][INFO ][o.o.d.PeerFinder         ] [node_t3] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_t3}{15gcWzhAS3e7yp-88UHxfQ}{pCVUdUvES--Usb2aUyosJA}{127.0.0.1}{127.0.0.1:64859}{d}{shard_indexing_pressure_enabled=true}]
[2024-10-24T04:35:28,863][INFO ][o.o.p.PluginsService     ] [node_t0] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:28,954][INFO ][o.o.c.m.MetadataCreateIndexService] [node_t0] [index] creating index, cause [api], templates [], shards [3]/[1]
[2024-10-24T04:35:29,027][INFO ][o.o.p.PluginsService     ] [node_t2] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,027][INFO ][o.o.p.PluginsService     ] [node_t3] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,027][INFO ][o.o.p.PluginsService     ] [node_t1] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,523][INFO ][o.o.i.r.RecoverySourceHandler] [node_t1] [index][2][recover to node_t2] finalizing recovery took [7.5ms]
[2024-10-24T04:35:29,523][INFO ][o.o.i.r.RecoverySourceHandler] [node_t3] [index][1][recover to node_t1] finalizing recovery took [7.5ms]
[2024-10-24T04:35:29,523][INFO ][o.o.i.r.RecoverySourceHandler] [node_t2] [index][0][recover to node_t3] finalizing recovery took [7.5ms]
[2024-10-24T04:35:29,560][INFO ][o.o.c.r.a.AllocationService] [node_t0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[index][1], [index][2]]]).
[2024-10-24T04:35:29,603][INFO ][o.o.c.m.MetadataIndexStateService] [node_t0] closing indices [index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,641][INFO ][o.o.c.m.MetadataIndexStateService] [node_t0] completed closing of indices [index]
[2024-10-24T04:35:29,721][INFO ][o.o.p.PluginsService     ] [node_t1] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,721][INFO ][o.o.p.PluginsService     ] [node_t2] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,721][INFO ][o.o.p.PluginsService     ] [node_t3] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,789][INFO ][o.o.p.PluginsService     ] [node_t2] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,789][INFO ][o.o.p.PluginsService     ] [node_t1] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,789][INFO ][o.o.p.PluginsService     ] [node_t3] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:35:29,865][INFO ][o.o.c.r.a.AllocationService] [node_t0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[index][0], [index][1]]]).
[2024-10-24T04:35:29,961][INFO ][o.o.c.c.RareClusterStateIT] [testDisassociateNodesWhileShardInit] Initializing shards
[2024-10-24T04:35:29,963][INFO ][o.o.c.c.RareClusterStateIT] [testDisassociateNodesWhileShardInit] [[index][2], node[bpTysqApSV6NKNVTVxuCNA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=SZmwUrsiS7iXZHrJVGzhZw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]], [index][0], node[15gcWzhAS3e7yp-88UHxfQ], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=4VtMD9rMRnyCcPsvkzVnuQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]], [index][1], node[sIJZ0V2STR2kUV2TTe6nTg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=2XqDCygUS5CagofS9fWfPw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]]]
[2024-10-24T04:35:29,968][INFO ][o.o.c.c.RareClusterStateIT] [node_t0] removed the node sIJZ0V2STR2kUV2TTe6nTg
[2024-10-24T04:35:29,968][INFO ][o.o.c.c.RareClusterStateIT] [node_t0] shard [index][2], node[bpTysqApSV6NKNVTVxuCNA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=SZmwUrsiS7iXZHrJVGzhZw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]]
[2024-10-24T04:35:29,971][INFO ][o.o.c.s.MasterService    ] [node_t0] failed to execute cluster state update (on version: [18], uuid: [ybe2gjzUR4uDDmXmtPjo0A]) for [Tasks batched with key: org.opensearch.cluster.coordination.RareClusterStateIT and count: 1]
nodes: 
   {node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}, local, cluster-manager
   {node_t2}{bpTysqApSV6NKNVTVxuCNA}{j2G6tAb6QdeUUdroD2DLoQ}{127.0.0.1}{127.0.0.1:64844}{d}{shard_indexing_pressure_enabled=true}
   {node_t3}{15gcWzhAS3e7yp-88UHxfQ}{pCVUdUvES--Usb2aUyosJA}{127.0.0.1}{127.0.0.1:64859}{d}{shard_indexing_pressure_enabled=true}
   {node_t1}{sIJZ0V2STR2kUV2TTe6nTg}{7aCEwCzFRx6MIPU33XT2jg}{127.0.0.1}{127.0.0.1:64835}{d}{shard_indexing_pressure_enabled=true}
routing_table (version 14):
-- index [[index/jHME7g_rTtahztAaE2-ZHw]]
----shard_id [index][0]
--------[index][0], node[bpTysqApSV6NKNVTVxuCNA], [P], s[STARTED], a[id=MdDlqp82TsCUrT-IiRH7Dg]
--------[index][0], node[15gcWzhAS3e7yp-88UHxfQ], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=4VtMD9rMRnyCcPsvkzVnuQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]]
----shard_id [index][1]
--------[index][1], node[15gcWzhAS3e7yp-88UHxfQ], [P], s[STARTED], a[id=5S3Kk0UKR7aDaWh4czTkRw]
--------[index][1], node[sIJZ0V2STR2kUV2TTe6nTg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=2XqDCygUS5CagofS9fWfPw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]]
----shard_id [index][2]
--------[index][2], node[bpTysqApSV6NKNVTVxuCNA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=SZmwUrsiS7iXZHrJVGzhZw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]]
--------[index][2], node[sIJZ0V2STR2kUV2TTe6nTg], [P], s[STARTED], a[id=SVO2jPYIRwGmanDM7Gsdiw]

routing_nodes:
-----node_id[bpTysqApSV6NKNVTVxuCNA][V]
--------[index][0], node[bpTysqApSV6NKNVTVxuCNA], [P], s[STARTED], a[id=MdDlqp82TsCUrT-IiRH7Dg]
--------[index][2], node[bpTysqApSV6NKNVTVxuCNA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=SZmwUrsiS7iXZHrJVGzhZw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]]
-----node_id[15gcWzhAS3e7yp-88UHxfQ][V]
--------[index][1], node[15gcWzhAS3e7yp-88UHxfQ], [P], s[STARTED], a[id=5S3Kk0UKR7aDaWh4czTkRw]
--------[index][0], node[15gcWzhAS3e7yp-88UHxfQ], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=4VtMD9rMRnyCcPsvkzVnuQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]]
-----node_id[sIJZ0V2STR2kUV2TTe6nTg][V]
--------[index][2], node[sIJZ0V2STR2kUV2TTe6nTg], [P], s[STARTED], a[id=SVO2jPYIRwGmanDM7Gsdiw]
--------[index][1], node[sIJZ0V2STR2kUV2TTe6nTg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=2XqDCygUS5CagofS9fWfPw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]]
---- unassigned

java.lang.NullPointerException: null
	at org.opensearch.gateway.ReplicaShardAllocator.cancelExistingRecoveryForBetterMatch(ReplicaShardAllocator.java:107) ~[classes/:?]
	at org.opensearch.gateway.ReplicaShardBatchAllocator.processExistingRecoveries(ReplicaShardBatchAllocator.java:77) ~[classes/:?]
	at org.opensearch.gateway.ShardsBatchGatewayAllocator.afterPrimariesBeforeReplicas(ShardsBatchGatewayAllocator.java:251) ~[classes/:?]
	at org.opensearch.cluster.routing.allocation.AllocationService.allocateAllUnassignedShards(AllocationService.java:627) ~[classes/:?]
	at org.opensearch.cluster.routing.allocation.AllocationService.allocateExistingUnassignedShards(AllocationService.java:598) ~[classes/:?]
	at org.opensearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:574) ~[classes/:?]
	at org.opensearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:542) ~[classes/:?]
	at org.opensearch.cluster.routing.allocation.AllocationService.disassociateDeadNodes(AllocationService.java:353) ~[classes/:?]
	at org.opensearch.cluster.coordination.RareClusterStateIT$5.execute(RareClusterStateIT.java:506) ~[classes/:?]
	at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:67) ~[classes/:?]
	at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:913) [classes/:?]
	at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:465) [classes/:?]
	at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:325) [classes/:?]
	at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:229) [classes/:?]
	at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:210) [classes/:?]
	at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:252) [classes/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:923) [classes/:?]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283) [classes/:?]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246) [classes/:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
[2024-10-24T04:35:29,990][INFO ][o.o.i.r.RecoverySourceHandler] [node_t1] [index][2][recover to node_t2] finalizing recovery took [6.8ms]
[2024-10-24T04:35:29,992][INFO ][o.o.i.r.RecoverySourceHandler] [node_t3] [index][1][recover to node_t1] finalizing recovery took [6.3ms]
[2024-10-24T04:35:29,992][INFO ][o.o.i.r.RecoverySourceHandler] [node_t2] [index][0][recover to node_t3] finalizing recovery took [6.1ms]
[2024-10-24T04:36:40,025][INFO ][o.o.c.c.RareClusterStateIT] [testDisassociateNodesWhileShardInit] [[index][2], node[bpTysqApSV6NKNVTVxuCNA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=SZmwUrsiS7iXZHrJVGzhZw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]], [index][0], node[15gcWzhAS3e7yp-88UHxfQ], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=4VtMD9rMRnyCcPsvkzVnuQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]], [index][1], node[sIJZ0V2STR2kUV2TTe6nTg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=2XqDCygUS5CagofS9fWfPw], unassigned_info[[reason=CLUSTER_RECOVERED], at[2024-10-24T04:35:29.743Z], delayed=false, allocation_status[no_attempt]]]
[2024-10-24T04:36:40,122][INFO ][o.o.p.PluginsService     ] [node_t1] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:36:40,122][INFO ][o.o.p.PluginsService     ] [node_t2] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:36:40,122][INFO ][o.o.p.PluginsService     ] [node_t3] PluginService:onIndexModule index:[index/jHME7g_rTtahztAaE2-ZHw]
[2024-10-24T04:36:40,190][INFO ][o.o.c.r.a.AllocationService] [node_t0] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[index][2]]]).
[2024-10-24T04:36:40,290][INFO ][o.o.i.r.RecoverySourceHandler] [node_t1] [index][2][recover to node_t2] finalizing recovery took [8.8ms]
[2024-10-24T04:36:40,290][INFO ][o.o.i.r.RecoverySourceHandler] [node_t2] [index][0][recover to node_t3] finalizing recovery took [7.6ms]
[2024-10-24T04:36:40,337][INFO ][o.o.i.r.RecoverySourceHandler] [node_t3] [index][1][recover to node_t1] finalizing recovery took [6.1ms]
[2024-10-24T04:36:40,346][INFO ][o.o.c.r.a.AllocationService] [node_t0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[index][2], [index][1]]]).
[2024-10-24T04:36:40,379][INFO ][o.o.c.c.RareClusterStateIT] [testDisassociateNodesWhileShardInit] after test
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2024-10-24T04:36:40,393][INFO ][o.o.t.OpenSearchTestClusterRule] [testDisassociateNodesWhileShardInit] [RareClusterStateIT#testDisassociateNodesWhileShardInit]: cleaning up after test
[2024-10-24T04:36:40,458][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_t0] [index/jHME7g_rTtahztAaE2-ZHw] deleting index
[2024-10-24T04:36:40,517][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] stopping ...
[2024-10-24T04:36:40,524][INFO ][o.o.c.c.FollowersChecker ] [testDisassociateNodesWhileShardInit] FollowerChecker{discoveryNode={node_t1}{sIJZ0V2STR2kUV2TTe6nTg}{7aCEwCzFRx6MIPU33XT2jg}{127.0.0.1}{127.0.0.1:64835}{d}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2024-10-24T04:36:40,524][INFO ][o.o.c.c.FollowersChecker ] [testDisassociateNodesWhileShardInit] FollowerChecker{discoveryNode={node_t3}{15gcWzhAS3e7yp-88UHxfQ}{pCVUdUvES--Usb2aUyosJA}{127.0.0.1}{127.0.0.1:64859}{d}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2024-10-24T04:36:40,524][INFO ][o.o.c.c.FollowersChecker ] [node_t0] FollowerChecker{discoveryNode={node_t1}{sIJZ0V2STR2kUV2TTe6nTg}{7aCEwCzFRx6MIPU33XT2jg}{127.0.0.1}{127.0.0.1:64835}{d}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2024-10-24T04:36:40,525][INFO ][o.o.c.c.FollowersChecker ] [testDisassociateNodesWhileShardInit] FollowerChecker{discoveryNode={node_t2}{bpTysqApSV6NKNVTVxuCNA}{j2G6tAb6QdeUUdroD2DLoQ}{127.0.0.1}{127.0.0.1:64844}{d}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2024-10-24T04:36:40,525][INFO ][o.o.c.c.FollowersChecker ] [node_t0] FollowerChecker{discoveryNode={node_t3}{15gcWzhAS3e7yp-88UHxfQ}{pCVUdUvES--Usb2aUyosJA}{127.0.0.1}{127.0.0.1:64859}{d}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2024-10-24T04:36:40,525][INFO ][o.o.c.c.FollowersChecker ] [node_t0] FollowerChecker{discoveryNode={node_t2}{bpTysqApSV6NKNVTVxuCNA}{j2G6tAb6QdeUUdroD2DLoQ}{127.0.0.1}{127.0.0.1:64844}{d}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty
[2024-10-24T04:36:40,527][INFO ][o.o.c.c.Coordinator      ] [node_t2] cluster-manager node [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [node_t0][127.0.0.1:64832][disconnected] disconnected
[2024-10-24T04:36:40,527][INFO ][o.o.c.c.Coordinator      ] [node_t1] cluster-manager node [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [node_t0][127.0.0.1:64832][disconnected] disconnected
[2024-10-24T04:36:40,527][INFO ][o.o.c.c.Coordinator      ] [node_t3] cluster-manager node [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [node_t0][127.0.0.1:64832][disconnected] disconnected
[2024-10-24T04:36:40,528][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] stopped
[2024-10-24T04:36:40,528][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] closing ...
[2024-10-24T04:36:40,528][INFO ][o.o.c.s.ClusterApplierService] [node_t1] cluster-manager node changed {previous [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 27, reason: becoming candidate: onLeaderFailure
[2024-10-24T04:36:40,528][INFO ][o.o.c.s.ClusterApplierService] [node_t2] cluster-manager node changed {previous [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 27, reason: becoming candidate: onLeaderFailure
[2024-10-24T04:36:40,529][INFO ][o.o.c.s.ClusterApplierService] [node_t3] cluster-manager node changed {previous [{node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true}], current []}, term: 1, version: 27, reason: becoming candidate: onLeaderFailure
[2024-10-24T04:36:40,530][WARN ][o.o.c.NodeConnectionsService] [node_t3] failed to connect to {node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [node_t0][127.0.0.1:64832] connect_exception
	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1106) ~[classes/:?]
	at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[classes/:?]
	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[classes/:?]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?]
	at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[classes/:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:160) ~[classes/:?]
	at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[classes/:?]
	at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[classes/:?]
	at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[classes/:?]
	at org.opensearch.nio.NioSelector.processKey(NioSelector.java:255) ~[classes/:?]
	at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[classes/:?]
	at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[classes/:?]
	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:157) ~[classes/:?]
	... 7 more
[2024-10-24T04:36:40,531][WARN ][o.o.c.NodeConnectionsService] [node_t1] failed to connect to {node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [node_t0][127.0.0.1:64832] connect_exception
	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1106) ~[classes/:?]
	at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[classes/:?]
	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[classes/:?]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?]
	at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[classes/:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:160) ~[classes/:?]
	at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[classes/:?]
	at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[classes/:?]
	at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[classes/:?]
	at org.opensearch.nio.NioSelector.processKey(NioSelector.java:255) ~[classes/:?]
	at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191) ~[classes/:?]
	at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[classes/:?]
	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:157) ~[classes/:?]
	... 7 more
[2024-10-24T04:36:40,530][WARN ][o.o.c.NodeConnectionsService] [node_t2] failed to connect to {node_t0}{wno7Y3oFSmW10pPq5Fkn8g}{kug2E83pRzOPiJ74i0EZ1Q}{127.0.0.1}{127.0.0.1:64832}{m}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [node_t0][127.0.0.1:64832] connect_exception
	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1106) ~[classes/:?]
	at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[classes/:?]
	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[classes/:?]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?]
	at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[classes/:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:160) ~[classes/:?]
	at org.opensearch.nio.EventHandler.handleConnect(EventHandler.java:130) ~[classes/:?]
	at org.opensearch.transport.nio.TestEventHandler.handleConnect(TestEventHandler.java:139) ~[classes/:?]
	at org.opensearch.nio.NioSelector.attemptConnect(NioSelector.java:446) ~[classes/:?]
	at org.opensearch.nio.NioSelector.registerChannel(NioSelector.java:469) ~[classes/:?]
	at org.opensearch.nio.NioSelector.setUpNewChannels(NioSelector.java:458) ~[classes/:?]
	at org.opensearch.nio.NioSelector.preSelect(NioSelector.java:279) ~[classes/:?]
	at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:172) ~[classes/:?]
	at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148) ~[classes/:?]
	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?]
	at org.opensearch.nio.SocketChannelContext.connect(SocketChannelContext.java:157) ~[classes/:?]
	... 9 more
[2024-10-24T04:36:40,534][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] closed
[2024-10-24T04:36:40,534][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] stopping ...
[2024-10-24T04:36:40,536][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] stopped
[2024-10-24T04:36:40,536][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] closing ...
[2024-10-24T04:36:40,538][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] closed
[2024-10-24T04:36:40,539][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] stopping ...
[2024-10-24T04:36:40,540][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] stopped
[2024-10-24T04:36:40,540][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] closing ...
[2024-10-24T04:36:40,542][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] closed
[2024-10-24T04:36:40,542][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] stopping ...
[2024-10-24T04:36:40,543][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] stopped
[2024-10-24T04:36:40,543][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] closing ...
[2024-10-24T04:36:40,545][INFO ][o.o.n.Node               ] [testDisassociateNodesWhileShardInit] closed
[2024-10-24T04:36:40,546][INFO ][o.o.t.OpenSearchTestClusterRule] [testDisassociateNodesWhileShardInit] [RareClusterStateIT#testDisassociateNodesWhileShardInit]: cleaned up after test

Process finished with exit code 0

Copy link
Contributor

❌ Gradle check result for 83a5525: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@rajiv-kv rajiv-kv changed the title rare reroute with npe Fix for reroute operation during node-left Oct 24, 2024
Copy link
Contributor

❌ Gradle check result for 4227892: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for c506262: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@rajiv-kv
Copy link
Contributor Author

❌ Gradle check result for c506262: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Flaky Test
14291
15944

Copy link
Contributor

❌ Gradle check result for 3c2bd1d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@rajiv-kv rajiv-kv force-pushed the reroute_npe branch 2 times, most recently from 844b95d to 0bfcc63 Compare October 28, 2024 12:40
@shwetathareja shwetathareja added skip-changelog backport 2.x Backport to 2.x branch labels Oct 28, 2024
Copy link
Contributor

❌ Gradle check result for 0bfcc63: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

✅ Gradle check result for bc1edaf: SUCCESS

@shwetathareja shwetathareja merged commit 9f7d3b6 into opensearch-project:main Oct 28, 2024
36 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 28, 2024
Signed-off-by: Rajiv Kumar Vaidyanathan <[email protected]>
(cherry picked from commit 9f7d3b6)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@dbwiddis
Copy link
Member

dbwiddis commented Nov 1, 2024

@rajiv-kv @shwetathareja why did we skip a change log entry for this bug fix?

@rajiv-kv
Copy link
Contributor Author

rajiv-kv commented Nov 1, 2024

@rajiv-kv @shwetathareja why did we skip a change log entry for this bug fix?

There wasn't any user-facing changes as the null check already exists in the code.

@dbwiddis
Copy link
Member

dbwiddis commented Nov 1, 2024

@rajiv-kv @shwetathareja why did we skip a change log entry for this bug fix?

There wasn't any user-facing changes as the null check already exists in the code.

But it's fixing a bug. (I think? I don't see a linked bug issue.)

Let me make sure I understand what's happening here: This was a node that was already shutting down, encountering an exception while shutting down? Thus the only impact would be confusing entries in logging in case anyone was trying to figure out why the node dropped in the first place?

In which case, while an end user may not see it, anyone trying to diagnose a node drop will see a change in log behavior.

dbwiddis pushed a commit that referenced this pull request Nov 2, 2024
…16507)

(cherry picked from commit 9f7d3b6)

Signed-off-by: Rajiv Kumar Vaidyanathan <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants