Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Spark Logs coming from Java from the Pipeline Logs #38

Open
thepushkarp opened this issue Sep 26, 2022 · 6 comments
Open

Remove Spark Logs coming from Java from the Pipeline Logs #38

thepushkarp opened this issue Sep 26, 2022 · 6 comments
Labels
good first issue Good for newcomers hacktoberfest Issue for Hacktoberfest Contributors

Comments

@thepushkarp
Copy link
Collaborator

On each pipeline run, when pySpark is initializing, it prints some logs on the CLI which are not particularly helpful for the user.

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

I am pretty sure these are not from the logger that we have initialized in python and are instead being printed from the background Java processes that Spark runs. We need to find a way to remove these from being printed.

@thepushkarp thepushkarp added good first issue Good for newcomers hacktoberfest Issue for Hacktoberfest Contributors labels Sep 26, 2022
@SaranshBaniyal
Copy link

@thepushkarp can you please assign this issue to me

@thepushkarp
Copy link
Collaborator Author

Thank you for your interest @SaranshBaniyal. Please feel free to create a PR.

@SaranshBaniyal
Copy link

@thepushkarp I am really sorry but I am unable to figure this issue out

@SaranshBaniyal
Copy link

you may unassign me

@SaranshBaniyal SaranshBaniyal removed their assignment Oct 12, 2022
@thepushkarp
Copy link
Collaborator Author

Yeah this is a tricky one @SaranshBaniyal
If you're still interested, let me know and we could work on this together.

@SaranshBaniyal
Copy link

yes, I am interested, but I don't know how to proceed on this. Can you help?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers hacktoberfest Issue for Hacktoberfest Contributors
Projects
None yet
Development

No branches or pull requests

2 participants