[FLINK-31923] Run nightly builds against multiple branches and Flink versions

[MartijnVisser]

This PR does a couple of things:

1. It syncs this CI workflow to use the shared CI tooling for externalized Flink connectors 907466c
2. It changes the nightly builds to support checking multiple branches and Flink combinations. 9f90f40
   For AWS specifically, that's:

• The v3.0 branch is tested against Flink 1.15.4 (latest available 1.15.* version) and Flink 1.16.2 (latest available 1.16.* version)
• The v4.1 branch is tested against Flink 1.16.2 (latest available 1.16.* version) and Flink 1.17.1 (latest available 1.17.* version)
• The main branch, since it's unreleased, is tested against Flink 1.16-SNAPSHOT, 1.17-SNAPSHOT and 1.18-SNAPSHOT

3. main is now also tested against 1.18-SNAPSHOT Removed this, since AWS doesn't compile against Flink 1.18-SNAPSHOT, see https://github.com/apache/flink-connector-aws/actions/runs/5276473761/jobs/9543236823?pr=80#step:13:7916

[hlteoh37]

Hmm, I can't seem to replicate the build error locally. Could we try re-running the checks?

Done this:

 mvn clean package -Dflink.version=1.17-SNAPSHOT

and

 mvn clean package -Dflink.version=1.16-SNAPSHOT

[dannycranmer]

I believe the failure is due to the Maven version not supporting the shading config. Similar to this PR #38

@z3d1k looked into this before and can probably provide more info.

[z3d1k]

As @dannycranmer mentioned, this is due flink-connector-kinesis module not working with shading in maven 3.8.6 and above.

Build failure against 1.18-SNAPSHOT is due to the backward incompatible change in architecture tests from FLINK-31804. This issue impacts multiple connector packages.

[MartijnVisser]

I believe the failure is due to the Maven version not supporting the shading config.

What's the plan to address this, given that Flink and other connectors now use a Maven 3.3+ version? Will you drop the old Kinesis connector when the new one is introduced, or fix the shading issues in the old version?

[dannycranmer]

I believe the failure is due to the Maven version not supporting the shading config.

What's the plan to address this, given that Flink and other connectors now use a Maven 3.3+ version? Will you drop the old Kinesis connector when the new one is introduced, or fix the shading issues in the old version?

We can either fix the shading issue now, or wait for the current source to become deprecated and deleted. I would be inclined to fix the current shading if possible. @z3d1k did you make any progress on a fix?

given that Flink and other connectors now use a Maven 3.3+ version

This connector also uses Maven 3.3+, it uses 3.8.5

[z3d1k]

@z3d1k did you make any progress on a fix?

No. I will take another look.