You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm working on a repro case, but looking at AWSGlueMetastoreCacheDecorator it seems to do nothing on dropTable? Shouldn't it invalidate the cache, else the next getTable() will falsely return it?
I'm working on a repro case, but looking at
AWSGlueMetastoreCacheDecorator
it seems to do nothing ondropTable
? Shouldn't it invalidate the cache, else the nextgetTable()
will falsely return it?I noticed this because I previously had Spark 2.4 + this lib and in cases where I do
spark.mode(overwrite).saveAsTable(foo)
, it was working fine. Now it fails in Spark 3.3 with the same code. Under the covers, overwrite is implemented as "dropTable, make sure does not exist, then createTable": https://github.com/apache/spark/blob/branch-3.3/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala#L673-L677cc @aws-jeffrey-yang
The text was updated successfully, but these errors were encountered: