Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CDAP-20758] Set max netty direct memory #15274

Merged
merged 1 commit into from
Aug 8, 2023
Merged

[CDAP-20758] Set max netty direct memory #15274

merged 1 commit into from
Aug 8, 2023

Conversation

rmstar
Copy link
Contributor

@rmstar rmstar commented Aug 4, 2023

In Spark v3.2.0 and later, max netty direct memory must be >= max Spark remote block fetch size (ref).

So we set netty's max direct memory = remote block fetch size.
Spark remote block fetch size was set to a large value to address some issues with joins as part of CDAP-16461.

This fixes the following executor pod failure:

2023-08-03 16:19:37,019 - ERROR [dispatcher-Executor:o.a.s.i.Logging@98] - Executor self-exiting due to : Unable to create executor due to Netty direct memory should at least be bigger than 'spark.network.maxRemoteBlockSizeFetchToMem', but got 2075918336 bytes < 2147483135
org.apache.spark.SparkException: Netty direct memory should at least be bigger than 'spark.network.maxRemoteBlockSizeFetchToMem', but got 2075918336 bytes < 2147483135
	at org.apache.spark.executor.CoarseGrainedExecutorBackend.onStart(CoarseGrainedExecutorBackend.scala:96)

@rmstar rmstar added the build Triggers github actions build label Aug 4, 2023
@rmstar rmstar requested a review from albertshau August 4, 2023 04:59
// In Spark v3.2.0 and later, max netty direct memory must be >= spark.network.maxRemoteBlockSizeFetchToMem
// See https://github.com/apache/spark/commit/00b63c8dc2170cfbc3f3ea7882dfd417d7fde744#.
// So we set max netty direct memory = spark.network.maxRemoteBlockSizeFetchToMem
// maxRemoteBlockSizeFetchToMem config is set in ETLSpark.java, DataStreamsSparkLauncher.java
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be good to reference the jira in case people want more context/info

It also seems like we should set it in the same place that spark.network.maxRemoteBlockSizeFetchToMem is being set. Otherwise if we ever change that value for some reason, this may break again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added reference to the jira. moved the code that sets io.netty.maxDirectMemory into the app where we set spark.network.maxRemoteBlockSizeFetchToMem

@rmstar rmstar merged commit 860f23f into develop Aug 8, 2023
4 checks passed
@rmstar rmstar deleted the CDAP20758 branch August 8, 2023 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Triggers github actions build
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants