[CELEBORN-1096] Avoid initializing SortShuffleManager when stop

### What changes were proposed in this pull request? ### Why are the changes needed? If there is no fallback to the Spark SortShuffleManager, it will be initialized once when stopping, and then stopped again. This is not necessary. ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Closes apache#2052 from cxzl25/CELEBORN-1096. Authored-by: sychen <[email protected]> Signed-off-by: Fu Chen <[email protected]>
kerwin-zk · Oct 30, 2023 · 70366ed · 70366ed
1 parent df40a28
commit 70366ed
Show file tree

Hide file tree

Showing 2 changed files with 5 additions and 4 deletions.
diff --git a/...nt-spark/spark-2/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java b/...nt-spark/spark-2/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
@@ -167,8 +167,9 @@ public void stop() {
     if (lifecycleManager != null) {
       lifecycleManager.stop();
     }
-    if (sortShuffleManager() != null) {
-      sortShuffleManager().stop();
+    if (_sortShuffleManager != null) {
+      _sortShuffleManager.stop();
+      _sortShuffleManager = null;
     }
   }
 

diff --git a/...nt-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java b/...nt-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
@@ -197,8 +197,8 @@ public void stop() {
       lifecycleManager.stop();
       lifecycleManager = null;
     }
-    if (sortShuffleManager() != null) {
-      sortShuffleManager().stop();
+    if (_sortShuffleManager != null) {
+      _sortShuffleManager.stop();
       _sortShuffleManager = null;
     }
   }