[#4867] feat(core): Add storage support for columns in Gravitino

[jerryshao]

What changes were proposed in this pull request?

This PR aims to add storage support for columns in Gravitino.

Why are the changes needed?

With this, we can also support managing columns in Gravitno, like set tags, do column level privileges.

Fix: #4867

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UTs added.

[yuqi1129]

Once we are going to release a new version then a corresponding schema SQL file is needed, am I right?

[jerryshao]

What is the meaning, can you please explain more?

[yuqi1129]

I wonder if the value of nullable is false, can we set the default value as null?

[jerryshao]

We use this DEFAULT_VALUE_NOT_SET to represent null, I think it is aligned with ColumnDTO's current implementation.

[mchades]

For column default value, we use DEFAULT_VALUE_NOT_SET and Literals.NULL to represent different default values：

• For DEFAULT_VALUE_NOT_SET, it means that the user has not set a default value. If the column is nullable=false, then this default value will be entirely determined by the underlying catalog (for example, some engines set the default value of column int not null to 0).
• For Literals.NULL, if the column nullable=true, we can use Literals.NULL to explicitly specify the default value as null. if we use DEFAULT_VALUE_NOT_SET, then this default value will be entirely determined by the underlying catalog

[jerryshao]

For column default value, we use DEFAULT_VALUE_NOT_SET and Literals.NULL to represent different default values：

• For DEFAULT_VALUE_NOT_SET, it means that the user has not set a default value. If the column is nullable=false, then this default value will be entirely determined by the underlying catalog (for example, some engines set the default value of column int not null to 0).
• For Literals.NULL, if the column nullable=true, we can use Literals.NULL to explicitly specify the default value as null. if we use DEFAULT_VALUE_NOT_SET, then this default value will be entirely determined by the underlying catalog

I see, let me update the code.

[yuqi1129]

Is there a good way to judge if there has been a change in the table so that we don't need to drop all columns and create them if no changes have been made to them such as table renaming, or altering table comments?

[yuqi1129]

TableColumnMetaService.getInstance().isColumnUpdated

[jerryshao]

we don't need to drop all columns and create them if no changes have been made to them such as table renaming, or altering table comments

If we don't want to update the columns, we have to fetch the old columns then do the comparison. The current overwrite doesn't fetch the old data, so we have to fetch again, the overhead is the same.

[yuqi1129]

if oldColumn == null, it means the newColumn is a newly created one, so the op type should not be update

[jerryshao]

It depends on how we define the UPDATE or CREATE, here in my implementation, I only set the type to CREATE when table is created, UPDATE for table update, no matter it is adding new column or updating the existing column.

Actually, no matter it is UPDATE or CREATE, what we only care if DELETE, which is used to filter out deleted columns.