Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-35868][cdc-connector][mongodb] Bump dependency version to support MongoDB 7.0 #3489

Merged
merged 5 commits into from
Jul 25, 2024
Merged

[FLINK-35868][cdc-connector][mongodb] Bump dependency version to support MongoDB 7.0 #3489

merged 5 commits into from
Jul 25, 2024

Conversation

yuxiqian
Copy link
Contributor

@yuxiqian yuxiqian commented Jul 23, 2024 •
edited
Loading

This closes FLINK-35868.

By bumping mongo-kafka and mongo-driver dependencies, Mongo CDC should be able to cooperate with MongoDB 6.0.16 & 7.0.12 (and legacy 5.0.2).

Notice: since JUnit doesn't allow parameterized @ClassRule or static fields, so containers must be created / destroyed before / after each test case, which significantly slows down the. testing (~50 min.). More discussion is required on this.

@yuxiqian yuxiqian marked this pull request as ready for review July 23, 2024 07:45
@yuxiqian
Copy link
Contributor Author

@leonardBang @Jiabao-Sun PTAL

@Jiabao-Sun Jiabao-Sun self-requested a review July 23, 2024 08:01
Jiabao-Sun
Jiabao-Sun approved these changes Jul 23, 2024
View reviewed changes
Copy link
Contributor

@Jiabao-Sun Jiabao-Sun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yuxiqian for this improvement.
LGTM.

@Jiabao-Sun Jiabao-Sun changed the title [FLINK-35868][mongo] Add MongoDB 6.0 & 7.0 tests [FLINK-35868][cdc-connector][mongodb] Add MongoDB 6.0 & 7.0 tests Jul 23, 2024
@Jiabao-Sun
Copy link
Contributor

Shall we bump the driver version from 4.7.1 to 5.1.1 as well?

@yuxiqian yuxiqian changed the title [FLINK-35868][cdc-connector][mongodb] Add MongoDB 6.0 & 7.0 tests [FLINK-35868][cdc-connector][mongodb] Bump dependency version to support MongoDB 7.0 Jul 23, 2024
@yuxiqian
Copy link
Contributor Author

Shall we bump the driver version from 4.7.1 to 5.1.1 as well?

Done, bumped mongo-kafka version, too.

leonardBang
leonardBang reviewed Jul 23, 2024
View reviewed changes
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mongodb-cdc/pom.xml Outdated
@@ -69,7 +69,7 @@ limitations under the License.
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongodb-driver-sync</artifactId>
<version>4.9.1</version>
<version>5.1.1</version>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also update the docs ？ and the older(release-3.1) version docs seem use an incorrect driver version.
https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/flink-sources/overview/

Jiabao-Sun
Jiabao-Sun requested changes Jul 23, 2024
View reviewed changes
Copy link
Contributor

@Jiabao-Sun Jiabao-Sun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI failure should be related to the mongo-kafka version upgrade.
We need to analyze the cause.

@yuxiqian
Copy link
Contributor Author

I'll investigate this.

@yuxiqian
Copy link
Contributor Author

yuxiqian commented Jul 23, 2024 •
edited
Loading

Seems this commit mongodb/mongo-java-driver@a03649c changed return type of listCollectionNames and could not work properly with mongo-kafka-connect. Seems it's not 100% safe to use 4.9.1 either, since mongo-kafka still sticks to 4.7.x as its dependency.

Any thoughts from @Jiabao-Sun?

@Jiabao-Sun
Copy link
Contributor

Thanks @yuxiqian for the investigation.
How about bump the driver version to 4.11?

@github-actions github-actions bot added the docs Improvements or additions to documentation label Jul 24, 2024
Jiabao-Sun
Jiabao-Sun approved these changes Jul 24, 2024
View reviewed changes
Copy link
Contributor

@Jiabao-Sun Jiabao-Sun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yuxiqian for this great work. LGTM now.
cc @leonardBang

leonardBang
leonardBang approved these changes Jul 24, 2024
View reviewed changes
Copy link
Contributor

@leonardBang leonardBang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1， Thanks @yuxiqian and @Jiabao-Sun for the nice work!

@leonardBang leonardBang merged commit 241eb03 into apache:master Jul 25, 2024
20 checks passed
@yuxiqian yuxiqian mentioned this pull request Jul 26, 2024
Closed
wuzhenhua01 pushed a commit to wuzhenhua01/flink-cdc-connectors that referenced this pull request Aug 4, 2024
@yuxiqian @wuzhenhua01
[FLINK-35868][cdc-connector][mongodb] Bump dependency version to supp…
af78d63 
…ort MongoDB 7.0

This closes apache#3489.
qiaozongmi pushed a commit to qiaozongmi/flink-cdc that referenced this pull request Sep 23, 2024
@yuxiqian @qiaozongmi
[FLINK-35868][cdc-connector][mongodb] Bump dependency version to supp…
5f4ff40 
…ort MongoDB 7.0

This closes apache#3489.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leonardBang leonardBang leonardBang approved these changes

@Jiabao-Sun Jiabao-Sun Jiabao-Sun approved these changes

Assignees
No one assigned
Labels
approved docs Improvements or additions to documentation e2e-tests mongodb-cdc-connector reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@yuxiqian @Jiabao-Sun @leonardBang