Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Yarn supports pyflink #2956

Merged
merged 3 commits into from
Sep 1, 2023
Merged

[Feature] Yarn supports pyflink #2956

merged 3 commits into from
Sep 1, 2023

Conversation

ChengJie1053
Copy link
Member

@ChengJie1053 ChengJie1053 commented Aug 21, 2023 •
edited
Loading

yarn application supports pyflink

issues: #2437

streampark pyflink currently supports yarn, k8s will be supported in the future

  1. Before using pyflink, users need to compile venv.zip and upload it to hdfs and $WORKSPACE/python(whether this can be optimized, directly compile venv.zip or other solutions), please refer to the document: https://github.com/apache/incubator-streampark/pull/2956/files#diff-abc9ac8d84878215b2c86971e3320fbe0a7333a9a6dde9eb82b6 583393792b40

2.yarn application mode can be extended by adding PythonOptions

3.yarn perjob mode can be extended by modifying getJobGraph

wangsizhu0504
wangsizhu0504 reviewed Aug 22, 2023
View reviewed changes
streampark-console/streampark-console-webapp/src/locales/lang/zh-CN/flink/app.ts
@@ -62,7 +62,7 @@ export default {
programJar: '程序Jar文件',
dynamicProperties: '动态参数',
hadoopConfigTitle: '系统 Hadoop',
dragUploadTitle: '单击或拖动 jar 到此区域以上传',
dragUploadTitle: '单击或拖动 jar或py 到此区域以上传',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is best to modify the English internationalization file synchronously (lang/en/flink/app.ts)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thank you for reviewing the code

@ChengJie1053 ChengJie1053 changed the title yarn application supports pyflink yarn supports pyflink Aug 22, 2023
@wolfboys
Copy link
Member

Thanks very much for your contribution. You can first initiate a proposal and email discussion. Apache Way attaches great importance to email discussions. It would be better to explain the relevant background and principles of this Pull Request clearly before initiating the PR.

@ChengJie1053
Copy link
Member Author

Thanks very much for your contribution. You can first initiate a proposal and email discussion. Apache Way attaches great importance to email discussions. It would be better to explain the relevant background and principles of this Pull Request clearly before initiating the PR.

Ok, thank you

@ChengJie1053 ChengJie1053 changed the title yarn supports pyflink [Feature] Yarn supports pyflink Aug 28, 2023
wolfboys
wolfboys approved these changes Sep 1, 2023
View reviewed changes
Copy link
Member

@wolfboys wolfboys left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wolfboys wolfboys merged commit e807c0e into apache:dev Sep 1, 2023
8 checks passed
saLeox pushed a commit to saLeox/incubator-streampark that referenced this pull request Sep 4, 2023
@ChengJie1053 @saLeox
[Feature] Yarn supports pyflink (apache#2956)
09ae6b9 
* yarn application supports pyflink

* yarn perjob supports pyflink

* pyflink supports flink connector
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wangsizhu0504 wangsizhu0504 wangsizhu0504 left review comments

@wolfboys wolfboys wolfboys approved these changes

Assignees
No one assigned
Labels
BACKEND BUILD FLINK WEB UI
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ChengJie1053 @wolfboys @wangsizhu0504