Remove Spark Logs coming from Java from the Pipeline Logs

[thepushkarp]

On each pipeline run, when pySpark is initializing, it prints some logs on the CLI which are not particularly helpful for the user.

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

I am pretty sure these are not from the logger that we have initialized in python and are instead being printed from the background Java processes that Spark runs. We need to find a way to remove these from being printed.

[SaranshBaniyal]

@thepushkarp can you please assign this issue to me

[thepushkarp]

Thank you for your interest @SaranshBaniyal. Please feel free to create a PR.

[SaranshBaniyal]

@thepushkarp I am really sorry but I am unable to figure this issue out

[SaranshBaniyal]

you may unassign me

[thepushkarp]

Yeah this is a tricky one @SaranshBaniyal
If you're still interested, let me know and we could work on this together.

[SaranshBaniyal]

yes, I am interested, but I don't know how to proceed on this. Can you help?