[VL] Skip UTF-8 validation in JSON parsing

[PHILO-HE]

What changes were proposed in this pull request?

See discussion: #5253 (comment)

By default, simdjson lib enables UTF-8 validation to check input json string. However, Spark always disregards the UTF-8 encoding invalidity of input.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[PHILO-HE]

@kecookier, do you care about the UTF-8 encoding validity of JSON string? Do we just need to align with Spark's behavior? cc @wang-zhun

[wang-zhun]

@PHILO-HE We don't need UTF-8 validity, we just need to be consistent with Spark.
For example, get_json_object(concat('{"a": "1", "b": "', string(unhex(hex(129))), '"}'), '$.a') compared to Gluten returning null, Spark returning 1 is more in line with our expectations

[kecookier]

@PHILO-HE Thanks. Just keep it consistent with Spark.

[PHILO-HE]

Has opened one thread in velox to discuss this issue: facebookincubator/velox#10639. If velox accepts this change, we can close this pr.

[PHILO-HE]

@wang-zhun, can you validate this pr with your cases? We can merge this one if CI passes and no issue is on your side.

[wang-zhun]

@PHILO-HE Thank you, I’ve verified that this patch is effective for my scenario.

before

now

[PHILO-HE]

@kecookier, could you approve this pr if you have no comment?

[rui-mo]

Thanks.

[kecookier]

Thanks.