-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[enhancement](mysql compatible) add user and procs_priv tables to mysql db in all catalogs #33058
Conversation
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
d4953f7
to
73195aa
Compare
run buildall |
TPC-H: Total hot run time: 38700 ms
|
TPC-DS: Total hot run time: 183902 ms
|
ClickBench: Total hot run time: 29.52 s
|
TeamCity be ut coverage result: |
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
|
SCH_WORKLOAD_GROUPS("WORKLOAD_GROUPS", "WORKLOAD_GROUPS", TSchemaTableType.SCH_WORKLOAD_GROUPS); | ||
SCH_WORKLOAD_GROUPS("WORKLOAD_GROUPS", "WORKLOAD_GROUPS", TSchemaTableType.SCH_WORKLOAD_GROUPS), | ||
SCHE_USER("user", "user", TSchemaTableType.SCH_USER), | ||
SCH_PROCS_PRIV("procs_priv", "procs_priv", TSchemaTableType.SCH_PROCS_PRIV); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where is procs_priv's schema scanner?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
procs_priv does not need a scanner for the time being, just use SchemaDummyScanner to read empty data.
@@ -1937,4 +1938,97 @@ public String getDefaultCloudCluster(String user) { | |||
return cluster; | |||
} | |||
// ====== END CLOUD ====== | |||
|
|||
// for mysql.user table | |||
public List<List<String>> getAllUserInfo() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
UserInfo is hard code data, why not fill it in be and avoid rpc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because the data in userinfo requires permissions and password policies, FE can better directly determine the data that needs to be sent.
@@ -409,6 +409,8 @@ protected List<String> filterDatabaseNames(List<String> remoteDbNames) { | |||
protected Set<String> getFilterInternalDatabases() { | |||
return ImmutableSet.<String>builder() | |||
.add("information_schema") | |||
.add("performance_schema") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why filter performance_schema?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do not need to obtain MySQL's performance_schema in Doris, which will increase the burden of obtaining metadata, so filtering
Please add regression test to select data from user/procs_priv, and show their schema. |
|
||
import java.util.List; | ||
|
||
public class ExternalMysqlDatabase extends ExternalDatabase { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1 How to use exeternal mysql database?
2 Can regression test be added to it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When creating ExternalCatalog, a database named mysql will be automatically created. This is to be compatible with some BI tools. It is used in ExternalCatalog.java. Unit tests and regression tests have been added to it.
This part of the test will be added later. I will obtain the statements sent when connecting to Doris from various BI or Database IDE tools and add them to the regression test. |
…ql db in all catalogs
run buildall |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
clang-tidy made some suggestions
{"password_policy.failed_login_counter", TYPE_VARCHAR, sizeof(StringRef), false}, | ||
{"password_policy.lock_time", TYPE_VARCHAR, sizeof(StringRef), false}}; | ||
|
||
SchemaUserScanner::SchemaUserScanner() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
be/src/exec/schema_scanner/schema_user_scanner.cpp:66:
- : SchemaScanner(_s_user_columns, TSchemaTableType::SCH_USER) {}
+ : SchemaScanner(_s_user_columns, TSchemaTableType::SCH_USER) = default;
TPC-H: Total hot run time: 38800 ms
|
TPC-DS: Total hot run time: 182168 ms
|
TeamCity be ut coverage result: |
ClickBench: Total hot run time: 30.68 s
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
…ql db in all catalogs (#33058) Issue Number: close #xxx This PR aims to enhance the compatibility of BI tools (such as Dbeaver, DataGrip) when using the mysql connector to connect to Doris, because some BI tools query some tables in the mysql database. In our tests, the user and procs_priv tables were mainly queried. This PR adds these two tables and adds actual data to the user table. However, please note that most of the fields in the user table are in Doris' own format rather than mysql format, so it can only ensure that the BI tool is querying No error is reported when accessing these tables, which does not guarantee that the data is completely displayed, and the tables under Doris's mysql database do not support data modification. Thanks to @liujiwen-up for assisting in testing
…ql db in all catalogs (#33058) Issue Number: close #xxx This PR aims to enhance the compatibility of BI tools (such as Dbeaver, DataGrip) when using the mysql connector to connect to Doris, because some BI tools query some tables in the mysql database. In our tests, the user and procs_priv tables were mainly queried. This PR adds these two tables and adds actual data to the user table. However, please note that most of the fields in the user table are in Doris' own format rather than mysql format, so it can only ensure that the BI tool is querying No error is reported when accessing these tables, which does not guarantee that the data is completely displayed, and the tables under Doris's mysql database do not support data modification. Thanks to @liujiwen-up for assisting in testing
Proposed changes
Issue Number: close #xxx
This PR aims to enhance the compatibility of BI tools (such as Dbeaver, DataGrip) when using the mysql connector to connect to Doris, because some BI tools query some tables in the mysql database. In our tests, the user and procs_priv tables were mainly queried. This PR adds these two tables and adds actual data to the user table. However, please note that most of the fields in the user table are in Doris' own format rather than mysql format, so it can only ensure that the BI tool is querying No error is reported when accessing these tables, which does not guarantee that the data is completely displayed, and the tables under Doris's mysql database do not support data modification.
Thanks to @liujiwen-up for assisting in testing
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...