Any plans to publish TiDB Flink connector to Maven Central Repository?

[kylemeow]

Hi there,

Integrating TiDB with Flink is a great idea, as Flink is superb in real-time computing and TiDB is suitable for quick data access. However, I found that the connector is only found in this GitHub repo and not uploaded to Maven repo yet, which is not convenient for project setup and future maintenance.

May I know if there are any plans to publish TiDB Flink connector to Maven repo?

Thanks : )

[sunxiaoguang]

Hi @kylemeow, thanks for bring up this issue. We can have some discussion about the maturity of this project and have a plan for publishing binary packages to maven repository.

[tisonkun]

Hi @kylemeow @sunxiaoguang , distributing our artifacts on Maven Central Repository is a good idea for general access for our works. I would like to list out the submodules under our TiBigData repository so far for an overall insight.

• io.tidb:tidb-jdbc, not depend on other submodules.
• io.tidb:bigdata-core, depends on io.tidb:tidb-jdbc.
• io.tidb:prestodb-connector, depends on io.tidb:bigdata-core and thus io.tidb:tidb-jdbc.
• io.tidb:prestosql-connector, depends on io.tidb:bigdata-core and thus io.tidb:tidb-jdbc.
• io.tidb:flink-tidb-connector, empty parent module for all flink-tidb-connector.
• io.tidb:flink-tidb-connector-base, dependes on io.tidb:bigdata-core and thus io.tidb:tidb-jdbc.
• io.tidb:flink-tidb-connector-1.11, dependes on io.tidb:flink-tidb-connector-base and thus io.tidb:bigdata-core and io.tidb:tidb-jdbc.
• io.tidb:flink-tidb-connector-1.12, dependes on io.tidb:flink-tidb-connector-base and thus io.tidb:bigdata-core and io.tidb:tidb-jdbc.

I think both module can be distributed separated, but we possibly mess different concept under TiBigData. The name is quite common which is then error-prone because we tend to put everything under a "common" repository.

JDBC needs not to be coupled with BigData and it is totally self-contained, I suggest it has its own place tidb-jdbc like clickhouse-jdbc.

io.tidb:bigdata-core depends on JDBC just for TIDB_PREFIX & MYSQL_PREFIX. It seems overkilled.

presto & flink can also have their own names as tidb-presto & tidb-flink respectively.

However, as an early stage of development, it is ok to keep all artifacts in one repository but keep in mind prevent unexpected dependency.

In this part my suggestion is about the published artifact name.

• flink-tidb-connector looks good.
• tidb-jdbc looks good.
• bigdata-core looks weird, it seems like a Java thin wrapper on the top of tikv client-java, maybe name it a tidb-java or so.
• prestodb-connector and prestorsql-connector is better to be prestodb-tidb-connector and prestosql-tidb-connector respectively.

and it could be better we sync the name of directory and module.

[sunxiaoguang]

Sure, let's have a thorough discussion over biweekly meeting about this after the holiday.

[marsishandsome]

i will push a v0.0.3 release to maven repo today.

[marsishandsome]

v0.0.3 is pushed to sonatype and will be synced to Maven Central in a few days.

https://oss.sonatype.org/service/local/repositories/releases/content/io/tidb/

https://repo1.maven.org/maven2/io/tidb/

[kylemeow]

Wow, glad to see it published on Maven. Nice work : )

[marsishandsome]

@tisonkun good suggestion.
As discussed with @sunxiaoguang , currently we will continue to keep all the components in one repo. Reasons are as follows:

• one repo is simple for fast-developing
• it's an incubator project (graduated submodules may move to a separated repo in the future)

[tisonkun]

@marsishandsome Thanks! It makes sense for me. Hope we meet the requirements for graduation soon.

I think we can close this issue because we do publish our code to Maven Central Repository.

But a light document about how to achieve it is admirable. We can carefully fade out anything private or secret.

Here is an example from Apache ZooKeeper community and of course we can be much shorter.

• https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=135860428