New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor the lifecycle control of flink application mode -dosubmit #2994

Merged

Al-assad merged 22 commits into apache:dev from caicancai:Refactor_the_lifecycle_control_of_Flink_application-mode_

Sep 7, 2023

Member

caicancai commented Aug 29, 2023 •

edited

Loading

What changes were proposed in this pull request

Issue Number: close #xxx

Brief change log

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).
KubernetesNativeApplicationClient.doSubmit 适配到 FlinkOperator. deployApplicationJob;

(or)

This change added tests and can be verified as follows:

Does this pull request potentially affect one of the following parts

Dependencies (does it add or upgrade a dependency): (yes / no)
yes

caicancai and others added 3 commits

August 27, 2023 17:20


          remove scala2.11

f3cf1fb


          [Feature] [Flink-K8s-V2] Refactor the lifecycle control of Flink appl…

f34315d

…ication-mode_dosubmit


          Merge branch 'apache:dev' into Refactor_the_lifecycle_control_of_Flin…

7980d71

…k_application-mode_

github-actions bot added the FLINK label

caicancai added 2 commits

August 29, 2023 12:59


          add v2 denpendcy

6fcfdf3


          Merge remote-tracking branch 'origin/Refactor_the_lifecycle_control_o…

17ecd9b

…f_Flink_application-mode_' into Refactor_the_lifecycle_control_of_Flink_application-mode_

github-actions bot added the BUILD label

wolfboys reviewed

View reviewed changes

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

caicancai added 2 commits

August 29, 2023 13:45


           mvn spotless:apply


          throw exception

e11491e

Member Author

caicancai commented Aug 29, 2023

Al-assad requested changes

View reviewed changes

Member

Al-assad left a comment

It looks like this part of the code doesn't actually call FlinkOperator.deployApplicationJob, which calls V2, and the actual runtime logic is still V1's job submission code

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated

+              //      clusterClient = clusterDescriptor
+              //        .deployApplicationCluster(clusterSpecification, applicationConfig)
+              //        .getClusterClient
+                    FlinkK8sOperator.deployApplicationJob(submitRequest.id, spec)

Member

Al-assad Aug 29, 2023

FlinkK8sOperator.deployApplicationJob is an IO strucuture, need to invoke .runIO to run it actually.

Member Author

caicancai Aug 30, 2023

@Al-assad This module calls deployApplicationJob api and reports an error

Member Author

caicancai Aug 30, 2023 •

edited

Loading

@Al-assad At present, it seems that the client module will throw an exception when calling the FlinkK8sOperator.deployApplication Job(submit Request.id, spec) API.

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

caicancai and others added 2 commits

August 30, 2023 10:44


          optimization

32076c8


          Merge branch 'apache:dev' into Refactor_the_lifecycle_control_of_Flin…

d2b3c7a

…k_application-mode_

Al-assad requested changes

View reviewed changes

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated

		@@ -62,14 +67,16 @@ object KubernetesNativeApplicationClient extends KubernetesNativeClientTrait {
		var clusterDescriptor: KubernetesClusterDescriptor = null

Member

Al-assad Sep 1, 2023

remove this code

Member Author

caicancai Sep 1, 2023

done

Member Author

caicancai Sep 1, 2023

@Al-assad Hello, thank you for your review, I think this piece is deleted again when the code is finished

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

caicancai and others added 2 commits

September 1, 2023 17:50


          Merge branch 'apache:dev' into Refactor_the_lifecycle_control_of_Flin…

09f1e2d

…k_application-mode_


          Code optimization

7e32072

wuchunfu reviewed

View reviewed changes

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

Member Author

caicancai commented Sep 1, 2023

Member Author

caicancai commented Sep 2, 2023

caicancai and others added 4 commits

September 3, 2023 11:21


          add comment

0ecf7c1


          add comment

8854af1


          Merge branch 'apache:dev' into Refactor_the_lifecycle_control_of_Flin…

dc88377

…k_application-mode_


          add zio denpency

c3b20a4

Al-assad requested changes

View reviewed changes

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

...c/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient.scala Outdated Show resolved Hide resolved

caicancai and others added 5 commits

September 5, 2023 16:50


          resolve dependency conflicts

4e2caa9


          add KubernetesNativeApplicationClient_V2

b69df1e


          Merge branch 'apache:dev' into Refactor_the_lifecycle_control_of_Flin…

2eb3510

…k_application-mode_


          Parameter optimization and add license

a5f81bc


          Parameter optimization and add license

34bb149

Member Author

caicancai commented Sep 6, 2023


          add flinkconfig param

10c52b4

Member Author

caicancai commented Sep 7, 2023

@Al-assad The response returns some null values for the time being, and it will be improved after discussing with people in the community

Al-assad requested changes

View reviewed changes

...ain/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient_V2.scala Outdated Show resolved Hide resolved


          Rename to KubernetesNativeApplicationV2

3ab44ae

Member Author

caicancai commented Sep 7, 2023

Al-assad approved these changes

View reviewed changes

Member

Al-assad left a comment

It seems that there are still some issues, I will submit a correction PR later.

...ain/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient_V2.scala Outdated Show resolved Hide resolved

...ain/scala/org/apache/streampark/flink/client/impl/KubernetesNativeApplicationClient_V2.scala Outdated Show resolved Hide resolved

Al-assad merged commit 8ff30e3 into apache:dev

4 of 6 checks passed

Al-assad mentioned this pull request

[Feat][Flink-K8s-V2] Refactor the lifecycle control of Flink K8s application-mode jobs #3037

Merged

caicancai deleted the Refactor_the_lifecycle_control_of_Flink_application-mode_ branch

September 11, 2023 01:50

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels