Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWSGlueMetastoreCacheDecorator needs to invalidate on delete #70

Open
MasterDDT opened this issue Aug 12, 2023 · 0 comments
Open

AWSGlueMetastoreCacheDecorator needs to invalidate on delete #70

MasterDDT opened this issue Aug 12, 2023 · 0 comments

Comments

@MasterDDT
Copy link

MasterDDT commented Aug 12, 2023

I'm working on a repro case, but looking at AWSGlueMetastoreCacheDecorator it seems to do nothing on dropTable? Shouldn't it invalidate the cache, else the next getTable() will falsely return it?

I noticed this because I previously had Spark 2.4 + this lib and in cases where I do spark.mode(overwrite).saveAsTable(foo), it was working fine. Now it fails in Spark 3.3 with the same code. Under the covers, overwrite is implemented as "dropTable, make sure does not exist, then createTable": https://github.com/apache/spark/blob/branch-3.3/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L673-L677

cc @aws-jeffrey-yang

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant