Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] DataFilters of scan contain Chinese fields cause scan fallback #7282

Closed
zml1206 opened this issue Sep 19, 2024 · 2 comments · Fixed by #7330
Closed

[VL] DataFilters of scan contain Chinese fields cause scan fallback #7282

zml1206 opened this issue Sep 19, 2024 · 2 comments · Fixed by #7330
Labels
bug Something isn't working triage

Comments

@zml1206
Copy link
Contributor

zml1206 commented Sep 19, 2024

Backend

VL (Velox)

Bug description

It was supported before, seems not supported after #6713.

spark.range(10).toDF("中文").write.parquet("/tmp/t1")
spark.read.parquet("/tmp/t1").filter("`中文`>1").show()
E20240919 17:24:07.829043  7309 Exceptions.h:67] Line: /incubator-gluten/ep/build-velox/build/velox_ep/velox/type/Tokenizer.cpp:208, Function:invalidSubfieldPath, Expression:  Invalid subfield path: 中文, Source: RUNTIME, ErrorCode: INVALID_STATE
24/09/19 17:24:07 WARN GlutenFallbackReporter: Validation failed for plan: Scan parquet [QueryId=0], due to: Native validation failed:
Optional[Validation failed due to exception caught at file:SubstraitToVeloxPlanValidator.cc line:1287 function:validate, thrown from file:Tokenizer.cpp line:208 function:invalidSubfieldPath, reason:Invalid subfield path: ].

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

@zml1206 zml1206 added bug Something isn't working triage labels Sep 19, 2024
@zml1206
Copy link
Contributor Author

zml1206 commented Sep 19, 2024

cc @rui-mo

@rui-mo
Copy link
Contributor

rui-mo commented Sep 23, 2024

@zml1206 Thanks for reporting. I will take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants