[CELEBORN-1026] Optimize registerShuffle fallback log

### What changes were proposed in this pull request? ### Why are the changes needed? According to apache#1955 , when celeborn is not available, `spark.dynamicAllocation.enabled=true` and `spark.shuffle.service. enabled=true`, shuffle data should not be lost. ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Closes apache#1963 from cxzl25/CELEBORN-1026. Authored-by: sychen <[email protected]> Signed-off-by: Shuang <[email protected]>
kerwin-zk · Oct 11, 2023 · 62ba44d · 62ba44d
1 parent 84a318f
commit 62ba44d
Showing 1 changed file with 2 additions and 1 deletion.
diff --git a/...nt-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java b/...nt-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
@@ -141,7 +141,8 @@ public <K, V, C> ShuffleHandle registerShuffle(
 
     if (fallbackPolicyRunner.applyAllFallbackPolicy(
         lifecycleManager, dependency.partitioner().numPartitions())) {
-      if (conf.getBoolean("spark.dynamicAllocation.enabled", false)) {
+      if (conf.getBoolean("spark.dynamicAllocation.enabled", false)
+          && !conf.getBoolean("spark.shuffle.service.enabled", false)) {
         logger.error(
             "DRA is enabled but we fallback to vanilla Spark SortShuffleManager for "
                 + "shuffle: {} due to fallback policy. It may cause block can not found when reducer "