forked from apache/incubator-streampark
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Feature] Yarn supports pyflink (apache#2956)
* yarn application supports pyflink * yarn perjob supports pyflink * pyflink supports flink connector
- Loading branch information
1 parent
2001a88
commit 09ae6b9
Showing
16 changed files
with
280 additions
and
16 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
41 changes: 41 additions & 0 deletions
41
streampark-common/src/main/scala/org/apache/streampark/common/util/EnvUtils.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.streampark.common.util; | ||
|
|
||
| import java.lang.reflect.Field; | ||
| import java.lang.reflect.Method; | ||
| import java.util.Map; | ||
|
|
||
|
|
||
| public class EnvUtils { | ||
| public static void setEnv(String name, String value) throws Exception { | ||
| getModifiableEnvironment().put(name, value); | ||
| } | ||
|
|
||
| @SuppressWarnings("unchecked") | ||
| private static Map<String, String> getModifiableEnvironment() throws Exception { | ||
| Class<?> pe = Class.forName("java.lang.ProcessEnvironment"); | ||
| Method getenv = pe.getDeclaredMethod("getenv"); | ||
| getenv.setAccessible(true); | ||
| Object unmodifiableEnvironment = getenv.invoke(null); | ||
| Class<?> map = Class.forName("java.util.Collections$UnmodifiableMap"); | ||
| Field m = map.getDeclaredField("m"); | ||
| m.setAccessible(true); | ||
| return (Map<String, String>) m.get(unmodifiableEnvironment); | ||
| } | ||
| } | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
87 changes: 87 additions & 0 deletions
87
...eampark-flink-client/streampark-flink-client-core/src/main/resources/pyflink.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
|
|
||
| ### 1. linux Creates Python virtual environments | ||
|
|
||
| #### 1.1. Prepare the 'setup-pyflink-virtual-env.sh script'. The default version of 'apache-flink' is '1.16.2'. You can change the version as required.The content is as follows | ||
|
|
||
| ```shell | ||
| set -e | ||
| # 下载Python 3.7 miniconda.sh脚本。 | ||
| wget "https://repo.continuum.io/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh" -O "miniconda.sh" | ||
|
|
||
| # 为Python 3.7 miniconda.sh脚本添加执行权限。 | ||
| chmod +x miniconda.sh | ||
|
|
||
| # 创建Python的虚拟环境。 | ||
| ./miniconda.sh -b -p venv | ||
|
|
||
| # 激活Conda Python虚拟环境。 | ||
| source venv/bin/activate "" | ||
|
|
||
| # 安装PyFlink依赖。 | ||
| pip install "apache-flink==1.16.2" | ||
|
|
||
| # 退出Conda Python虚拟环境。 | ||
| conda deactivate | ||
|
|
||
| # 删除缓存的包。 | ||
| rm -rf venv/pkgs | ||
|
|
||
| # 将准备好的Conda Python虚拟环境打包。 | ||
| zip -r venv.zip venv | ||
| ``` | ||
|
|
||
| #### 1.2. Prepare the 'build.sh script'. The content is as follows | ||
|
|
||
| The '/build' directory needs to be created by itself. You can also change it to another directory | ||
|
|
||
| ```shell | ||
| #!/bin/bash | ||
| set -e -x | ||
| yum install -y zip wget | ||
|
|
||
| cd /root/ | ||
| bash /build/setup-pyflink-virtual-env.sh | ||
| mv venv.zip /build/ | ||
| ``` | ||
|
|
||
| #### 1.3. Execute the following command | ||
|
|
||
| ```shell | ||
| docker run -it --rm -v $PWD:/build -w /build quay.io/pypa/manylinux2014_x86_64 ./build.sh | ||
| ``` | ||
| After this command is executed, a file named venv.zip will be generated, which is the virtual environment of Python 3.7. You can also modify the above script, install another version of the Python virtual environment, or install the required third-party Python packages in the virtual environment. | ||
|
|
||
| ### 2. Upload venv.zip to hdfs | ||
|
|
||
| venv.zip is about 539M in size and needs to be uploaded to hdfs by itself | ||
|
|
||
| ```shell | ||
| hadoop fs -put ./venv.zip /streampark/python | ||
| ``` | ||
|
|
||
| ### 3. Copy venv.zip to $WORKSPACE/python | ||
|
|
||
| ```shell | ||
| copy ./venv.zip $WORKSPACE/python | ||
| ``` | ||
|
|
||
| ### 4. Copy Python dependencies to $FLINK_HOME/lib | ||
|
|
||
| ```shell | ||
| cp -r $FLINK_HOME/opt/python $FLINK_HOME/lib | ||
|
|
||
| cp $FLINK_HOME/opt/flink-python-* $FLINK_HOME/lib | ||
| ``` | ||
|
|
||
| ### 5. If you use a flink connector dependency in your pyflink job, you need to put it in $FLINK_HOME/lib | ||
|
|
||
| ### 6. Reference document | ||
| ```text | ||
| https://help.aliyun.com/document_detail/413966.html#:~:text=.%2Fsetup-pyflink-%20virtual%20-env.sh,%E8%AF%A5%E5%91%BD%E4%BB%A4%E6%89%A7%E8%A1%8C%E5%AE%8C%E6%88%90%E5%90%8E%EF%BC%8C%E4%BC%9A%E7%94%9F%E6%88%90%E4%B8%80%E4%B8%AA%E5%90%8D%E4%B8%BA%20venv%20%E7%9A%84%E7%9B%AE%E5%BD%95%EF%BC%8C%E5%8D%B3%E4%B8%BAPython%203.6%E7%9A%84%E8%99%9A%E6%8B%9F%E7%8E%AF%E5%A2%83%E3%80%82 | ||
| https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/deployment/cli/#submitting-pyflink-jobs | ||
| https://nightlies.apache.org/flink/flink-docs-release-1.16/zh/docs/dev/python/python_config/ | ||
| ``` | ||
|
|
||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.