[VL] Refine setting shuffle writer local directories

[marin-ma]

Currently, shuffle writer local directories are setting through environment variable "GLUTEN_SPARK_LOCAL_DIRS", which is a historical issue and should be refined.

This PR removes setting/getting local directories through environment variable from JNI code path and keep the environment variable "GLUTEN_SPARK_LOCAL_DIRS" only for generic benchmark simulating spilling multiple disks.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[zhztheplayer]

Thanks!

[GlutenPerfBot]

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query	log/native_3601_time.csv	log/native_master_11_02_2023_78104be3e_time.csv	difference	percentage
q1	37.42	34.98	-2.440	93.48%
q2	25.18	24.86	-0.326	98.70%
q3	39.50	38.38	-1.123	97.16%
q4	37.03	37.54	0.513	101.39%
q5	69.27	71.16	1.889	102.73%
q6	8.82	7.29	-1.528	82.67%
q7	87.37	88.90	1.530	101.75%
q8	85.50	87.43	1.931	102.26%
q9	119.06	120.87	1.813	101.52%
q10	49.67	51.83	2.156	104.34%
q11	19.89	19.61	-0.278	98.60%
q12	28.27	26.28	-1.993	92.95%
q13	49.25	48.19	-1.059	97.85%
q14	19.52	18.42	-1.102	94.36%
q15	33.58	32.90	-0.686	97.96%
q16	16.17	15.99	-0.175	98.92%
q17	101.39	101.55	0.168	100.17%
q18	147.52	147.25	-0.278	99.81%
q19	18.35	17.00	-1.357	92.61%
q20	31.08	31.82	0.744	102.40%
q21	224.83	226.44	1.610	100.72%
q22	13.61	13.39	-0.216	98.41%
total	1262.28	1262.07	-0.207	99.98%