Skip to content

Commit

Permalink
Update README-ZH.md;Update config file path
Browse files Browse the repository at this point in the history
  • Loading branch information
jefftlin committed Aug 23, 2023
1 parent 7bb49cb commit a748093
Show file tree
Hide file tree
Showing 19 changed files with 125 additions and 38 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/dead-link-checker.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: Dead Link Check

on: [push, pull_request]
on: [push]

jobs:
dead-links-check:
Expand Down
15 changes: 9 additions & 6 deletions README-ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

## 介绍

Exchangis 1.1.2 是微众银行大数据平台 WeDataSphere 与社区用户共同研发的的新版数据交换工具,支持异构数据源之间的结构化和非结构化数据传输同步。
Exchangis是微众银行大数据平台 WeDataSphere 与社区用户共同研发的的新版数据交换工具,支持异构数据源之间的结构化和非结构化数据传输同步。

Exchangis 抽象了一套统一的数据源和同步作业定义插件,允许用户快速接入新的数据源,并只需在数据库中简单配置即可在页面中使用。

Expand All @@ -16,7 +16,7 @@ Exchangis 抽象了一套统一的数据源和同步作业定义插件,允许

### 界面预览

![image](https://user-images.githubusercontent.com/27387830/171488936-2cea3ee9-4ef7-4309-93e1-e3b697bd3be1.png)
![image](images/zh_CN/ch1/frontend_view.png)

## 核心特点

Expand All @@ -42,19 +42,22 @@ Exchangis 抽象了一套统一的数据源和同步作业定义插件,允许
### 3. 与DSS工作流打通,一站式大数据开发的门户

- 实现DSS AppConn包括一级 SSO 规范,二级组织结构规范,三级开发流程规范在内的三级规范;

- 作为DSS工作流的数据交换节点,是整个工作流链路中的门户流程,为后续的工作流节点运行提供稳固的数据基础;

### 4. 支持多种导数引擎

- 支持Sqoop和DataX引擎进行多种异构数据源之间的导数

## 整体设计

### 架构设计

![架构设计](https://user-images.githubusercontent.com/27387830/173026793-f1475803-9f85-4478-b566-1ad1d002cd8a.png)
![架构设计](images/zh_CN/ch1/home_page_zh.png)


## 相关文档
[安装部署文档](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.1.2/docs/zh_CN/ch1/exchangis_deploy_cn.md)
[用户手册](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.1.2/docs/zh_CN/ch1/exchangis_user_manual_cn.md)
[安装部署文档](docs/zh_CN/ch1/exchangis_deploy_cn.md)
[用户手册](docs/zh_CN/ch1/exchangis_user_manual_cn.md)

## 交流贡献

Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ English | [中文](README-ZH.md)

## Introduction

Exchangis 1.1.2 is a new version of data exchange tool jointly developed by WeDataSphere, a big data platform of WeBank, and community users, which supports the synchronization of structured and unstructured data transmission between heterogeneous data sources.
Exchangis is a new version of data exchange tool jointly developed by WeDataSphere, a big data platform of WeBank, and community users, which supports the synchronization of structured and unstructured data transmission between heterogeneous data sources.

Exchangis abstracts a unified set of data source and synchronization job definition plugins, allowing users to quickly access new data sources and use them on pages with simple configuration in the database.

Expand All @@ -14,7 +14,7 @@ With the help of [Linkis](https://github.com/apache/incubator-linkis) computing

### Interface preview

![image](https://user-images.githubusercontent.com/27387830/171488936-2cea3ee9-4ef7-4309-93e1-e3b697bd3be1.png)
![image](images/zh_CN/ch1/frontend_view.png)

## Core characteristics

Expand Down Expand Up @@ -47,13 +47,13 @@ With the help of [Linkis](https://github.com/apache/incubator-linkis) computing

### Architecture Design

![架构设计](images/en_US/ch1/architecture.png)
![架构设计](images/zh_CN/ch1/home_page_en.png)


## Documents

[Quick Deploy](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.1.2/docs/en_US/ch1/exchangis_deploy_en.md)
[User Manual](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.1.2/docs/en_US/ch1/exchangis_user_manual_en.md)
[Quick Deploy](docs/en_US/ch1/exchangis_deploy_en.md)
[User Manual](docs/en_US/ch1/exchangis_user_manual_en.md)

## Communication and contribution

Expand Down
4 changes: 2 additions & 2 deletions assembly-package/sbin/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ interact_echo(){

# Initalize database
init_database(){
BOOTSTRAP_PROP_FILE="${CONF_PATH}/exchangis-server.properties"
BOOTSTRAP_PROP_FILE="${CONF_PATH}/dss-exchangis-server.properties"
if [ "x${SQL_SOURCE_PATH}" != "x" ] && [ -f "${SQL_SOURCE_PATH}" ]; then
`mysql --version >/dev/null 2>&1`
DATASOURCE_URL="jdbc:mysql:\/\/${MYSQL_HOST}:${MYSQL_PORT}\/${DATABASE}\?useSSL=false\&characterEncoding=UTF-8\&allowMultiQueries=true"
Expand All @@ -142,7 +142,7 @@ init_database(){
}

init_properties(){
BOOTSTRAP_PROP_FILE="${CONF_PATH}/exchangis-server.properties"
BOOTSTRAP_PROP_FILE="${CONF_PATH}/dss-exchangis-server.properties"
APPLICATION_YML="${CONF_PATH}/application-exchangis.yml"
LINKIS_GATEWAY_URL="http:\/\/${LINKIS_GATEWAY_HOST}:${LINKIS_GATEWAY_PORT}\/"
if [ "x${LINKIS_SERVER_URL}" == "x" ]; then
Expand Down
2 changes: 1 addition & 1 deletion docs/en_US/ch1/component_upgrade_en.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,4 @@ After the installation and deployment of exchangis-appconn is completed, you can
2. Check whether the project is created synchronously on the exchange side. If the creation is successful, the appconn installation is successful.
![image](https://user-images.githubusercontent.com/27387830/169782337-678f2df0-080a-495a-b59f-a98c5a427cf8.png)

For more usage, please refer to[Exchangis1.0 User Manual](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.0.1-rc/docs/zh_CN/ch1/exchangis_user_manual_cn.md)
For more usage, please refer to[Exchangis User Manual](docs/zh_CN/ch1/exchangis_user_manual_cn.md)
4 changes: 2 additions & 2 deletions docs/en_US/ch1/exchangis_appconn_deploy_en.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This paper mainly introduces the deployment, configuration and installation of ExchangisAppConn in DSS(DataSphere Studio)1.0.1.

### 1. Preparations for the deployment of ExchangisAppConn
Before you deploy ExchangisAppConn, please follow the [Exchangis1.0.0 to install the deployment document](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.1.2/docs/en_US/ch1/exchangis_deploy_en.md) to complete the installation of Exchangis1.0.0 and other related components, and ensure that the basic functions of the project are available.
Before you deploy ExchangisAppConn, please follow the [Exchangis to install the deployment document](docs/en_US/ch1/exchangis_deploy_en.md) to complete the installation of Exchangis and other related components, and ensure that the basic functions of the project are available.

### 2. Download and compilation of the ExchangisAppConn plugin
#### 1) Download binary package
Expand Down Expand Up @@ -67,7 +67,7 @@ After the exchangis-appconn is installed and deployed, the following steps can b
2. Check whether the project is created synchronously on Exchangis. Successful creation means successful installation of appconn
![image](https://user-images.githubusercontent.com/27387830/169782337-678f2df0-080a-495a-b59f-a98c5a427cf8.png)

For more operation, please refer to [Exchangis 1.0 User Manual](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.0.0-rc/docs/zh_CN/ch1/exchangis_user_manual_cn.md)
For more operation, please refer to [Exchangis User Manual](docs/zh_CN/ch1/exchangis_user_manual_cn.md)

### 5.Exchangis AppConn installation principle

Expand Down
84 changes: 84 additions & 0 deletions docs/en_US/ch1/exchangis_datax_deploy_en.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# DataX engine uses documentation

### Prepare the environment

The DataX engine is an indispensable component for executing Exchangis data synchronization tasks. Data synchronization tasks can be performed only after the DataX engine is installed and deployed. Also, ensure that DataX is installed on the deployed machine.

Before you install and deploy DataX engine, Please complete the installation of Exchangis and related components according to the [Exchangis installation and deployment document](docs/en_US/ch1/exchangis_deploy_en.md), and ensure that the basic functions of the project are available.

It is strongly recommended that you use the native DataX to perform the test task on this node before performing the DataX task, so as to check whether the environment of this node is normal.

| Environment variable name | Environment variable content | Remark |
| :-----------------------: | :--------------------------: | ------------ |
| JAVA_HOME | JDK installation path | Required |
| DATAX_HOME | DataX installation path | Not Required |
| DATAX_CONF_DIR | DataX config path | Not Required |

### Prepare installation package

#### 1)Download binary package

Exchangis1.1.2 and Linkis 1.4.0 support the mainstream DataX versions 1.4.6 and 1.4.7, and later versions may need to modify some codes for recompilation.

[Click to jump to Release interface](https://github.com/WeBankFinTech/Exchangis/releases/tag/release-1.1.2)

#### 2)Compile and package

If you want to develop and compile datax engine yourself, the specific compilation steps are as follows:

1.clone Exchangis's source code

2.Under exchangis-plugins module, find sqoop engine and compile sqoop separately, as follows :

```
cd {EXCHANGIS_CODE_HOME}/exchangis-plugins/engine/datax
mvn clean install
```

Then the datax engine installation package will be found in this path.

```
{EXCHANGIS_CODE_HOME}\exchangis-plugins\datax\target\out\datax
```


### Start deployment

#### 1)DataX engine installation

1、Get the packed datax.zip material package, the directory structure is

```shell
datax
-- dist
-- plugin
```

2、Place in the following directory in the linkis installation path

```shell
cd {LINKIS_HOME}/linkis/lib/linkis-engineconn-plugins
```

(Note that depending on which users the current datax engine has permissions for, they are generally hadoop user groups and hadoop users)


#### 2)Restart linkis-engineplugin service to make datax engine take effect

New engines joining linkis will not take effect until the engineplugin service of linkis is restarted, and the restart script is. /linkis-daemon.sh in the Linkis installation directory. The specific steps are as follows :

```
cd {LINKIS_INSTALL_HOME}/links/sbin/
./linkis-daemon.sh restart cg-engineplugin
```

After the service is successfully started, check whether the datax engine is installed in the linkis database

```shell
select * from linkis_cg_engine_conn_plugin_bml_resources where engine_conn_type='datax';
```
At this point, the datax installation and deployment is complete.
For a more detailed introduction of engineplugin, please refer to the following article.
https://linkis.apache.org/zh-CN/docs/latest/deployment/install-engineconn
8 changes: 4 additions & 4 deletions docs/en_US/ch1/exchangis_deploy_en.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Please ensure that DSS1.1.2 and Linkis1.4.0 are basically available. HiveQL scri

#### 2.1.1 Download binary package

Download the latest installation package from the Released release of Exchangis [click to jump to the release interface](https://github.com/WeBankFinTech/Exchangis/releases/tag/release-1.1.2).
Download the latest installation package from the Released release of Exchangis [click to jump to the release interface](https://github.com/WeBankFinTech/Exchangis/releases).

#### 2.1.2 Compile and package

Expand Down Expand Up @@ -214,7 +214,7 @@ As shown in the figure below:

#### 2.7.1 Get the front-end installation package

Exchangis has provided compiled front-end installation package by default, which can be downloaded and used directly :[Click to jump to the Release interface](https://github.com/WeBankFinTech/Exchangis/releases/download/release-1.0.0-rc1/web-dist.zip)
Exchangis has provided compiled front-end installation package by default, which can be downloaded and used directly :[Click to jump to the Release interface](https://github.com/WeBankFinTech/Exchangis/releases)

You can also compile the exchange front-end by yourself and execute the following command in the exchanise root directory:

Expand Down Expand Up @@ -296,12 +296,12 @@ Please visit the Exchange front-end page at http://${EXCHANGIS_INSTALL_IP}:8098

## 3. DSS ExchangisAppConn installation and deployment

If you want to use Exchangis1.0.0 front-end, you also need to install the DSS ExchangisAppConn plugin. Please refer to: [ExchangisAppConn installation documentation for plugins ](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.0.0/docs/en_US/ch1/exchangis_appconn_deploy_en.md)
If you want to use Exchangis1.0.0 front-end, you also need to install the DSS ExchangisAppConn plugin. Please refer to: [ExchangisAppConn installation documentation for plugins ](docs/en_US/ch1/exchangis_appconn_deploy_en.md)

## 4. Linkis Sqoop engine installation and deployment

If you want to execute the Sqoop operation of Exchangis1.0.0 normally, you also need to install the Linkis Sqoop engine. Please refer to: : [Linkis Sqoop engine installation documentation ](https://linkis.apache.org/zh-CN/docs/1.1.2/engine-usage/sqoop/)

## 5. How to log in and use Exchangis

Exchangis1.0 for more instructions, please refer to the user manual.[Exchangis1.0 user manual](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.0.0/docs/en_US/ch1/exchangis_user_manual_en.md)
Exchangis1.0 for more instructions, please refer to the user manual.[Exchangis user manual](docs/en_US/ch1/exchangis_user_manual_en.md)
4 changes: 2 additions & 2 deletions docs/en_US/ch1/exchangis_sqoop_deploy_en.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
### Prepare the environment
Sqoop engine is an indispensable component to perform Exchange IS data synchronization task, and only after the installation and deployment of Sqoop engine can it successfully perform data synchronization task. At the same time, make sure sqoop is installed on the deployed machine.

Before you install and deploy Sqoop engine, Please complete the installation of Exchangis 1.1.2 and related components according to the [Exchangis 1.1.2 installation and deployment document](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.1.2/docs/en_US/ch1/exchangis_deploy_en.md), and ensure that the basic functions of the project are available.
Before you install and deploy Sqoop engine, Please complete the installation of Exchangis and related components according to the [Exchangis installation and deployment document](docs/en_US/ch1/exchangis_deploy_en.md), and ensure that the basic functions of the project are available.

Sqoop engine mainly depends on Hadoop basic environment. If this node needs to deploy Sqoop engine, it needs to deploy Hadoop client environment.

Expand All @@ -28,7 +28,7 @@ It is strongly recommended that you use the native Sqoop to perform the test tas

Exchangis1.1.2 and Linkis 1.4.0 support the mainstream Sqoop versions 1.4.6 and 1.4.7, and later versions may need to modify some codes for recompilation.

[Click to jump to Release interface](https://github.com/WeBankFinTech/Exchangis/releases/tag/release-1.0.0)
[Click to jump to Release interface](https://github.com/WeBankFinTech/Exchangis/releases/tag/release-1.1.2)

#### 2)Compile and package
If you want to develop and compile sqoop engine yourself, the specific compilation steps are as follows:
Expand Down
2 changes: 1 addition & 1 deletion docs/en_US/ch1/exchangis_user_manual_en.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

## 二、Login Exchangis1.0

   Exchangis1.0 is currently a part of DSS**data exchange component**, and it can be accessed in the component list by logging in to DSS. Therefore, before using Exchangis 1.0, please make basic deployment of DSS, Exchange IS 1.0, Linkis and other related components to ensure that the components' functions are available. This article will not go into details. See for details:[exchangis_deploy_en](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.0.0/docs/en_US/ch1/exchangis_deploy_en.md)[exchangis-appconn_deploy_en](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.0.0/docs/en_US/ch1/exchangis_appconn_deploy_en.md)
   Exchangis1.0 is currently a part of DSS**data exchange component**, and it can be accessed in the component list by logging in to DSS. Therefore, before using Exchangis 1.0, please make basic deployment of DSS, Exchange IS 1.0, Linkis and other related components to ensure that the components' functions are available. This article will not go into details. See for details:[exchangis_deploy_en](docs/en_US/ch1/exchangis_deploy_en.md)[exchangis-appconn_deploy_en](docs/en_US/ch1/exchangis_appconn_deploy_en.md)

### 1、Login DSS

Expand Down
6 changes: 3 additions & 3 deletions docs/zh_CN/ch1/component_upgrade_cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,9 @@ lib/exchangis-server
```shell script
./sbin/daemon.sh restart server
```
# 4.3)更新exchangis-server.propertis文件
# 4.3)更新dss-exchangis-server.propertis文件

将exchangis-server.propertis文件中的最后一行进行替换,替换内容如下
将dss-exchangis-server.propertis文件中的最后一行进行替换,替换内容如下

```$xslt
wds.linkis-session.ticket.key=bdp-user-ticket-id
Expand All @@ -94,4 +94,4 @@ wds.linkis-session.ticket.key=bdp-user-ticket-id
![image](https://user-images.githubusercontent.com/27387830/169782337-678f2df0-080a-495a-b59f-a98c5a427cf8.png)


更多使用操作可参照[Exchangis用户手册](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.1.2/docs/zh_CN/ch1/exchangis_user_manual_cn.md)
更多使用操作可参照[Exchangis用户手册](docs/zh_CN/ch1/exchangis_user_manual_cn.md)
6 changes: 3 additions & 3 deletions docs/zh_CN/ch1/exchangis_appconn_deploy_cn.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
ExchangisAppConn安装文档
本文主要介绍在DSS(DataSphere Studio)1.0.1中ExchangisAppConn的部署、配置以及安装
### 1.部署ExchangisAppConn的准备工作
您在部署ExchangisAppConn之前,请按照[Exchangis1.1.2安装部署文档](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.1.2/docs/zh_CN/ch1/exchangis_deploy_cn.md)安装完成Exchangis1.1.2及其他相关组件的安装,并确保工程基本功能可用。
您在部署ExchangisAppConn之前,请按照[Exchangis安装部署文档](docs/zh_CN/ch1/exchangis_deploy_cn.md)安装完成Exchangis及其他相关组件的安装,并确保工程基本功能可用。

### 2.ExchangisAppConn插件的下载和编译
#### 1)下载二进制包
我们提供ExchangisAppconn的物料包,您可直接下载使用。[点击跳转 Release 界面](https://osp-1257653870.cos.ap-guangzhou.myqcloud.com/WeDatasphere/Exchangis/exchangis1.0.0/exchangis-appconn.zip)
我们提供ExchangisAppconn的物料包,您可直接下载使用。[点击跳转 Release 界面](https://github.com/WeBankFinTech/Exchangis/releases)
#### 2) 编译打包

如果您想自己开发和编译ExchangisAppConn,具体编译步骤如下:
Expand Down Expand Up @@ -64,7 +64,7 @@ sh ./dss-start-all.sh
2. 在exchangis端查看是否同步创建项目,创建成功说明appconn安装成功
![image](https://user-images.githubusercontent.com/27387830/169782337-678f2df0-080a-495a-b59f-a98c5a427cf8.png)

更多使用操作可参照[Exchangis1.0用户手册](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.0.0/docs/zh_CN/ch1/exchangis_user_manual_cn.md)
更多使用操作可参照[Exchangis用户手册](docs/zh_CN/ch1/exchangis_user_manual_cn.md)

### 5.Exchangis AppConn安装原理

Expand Down
4 changes: 2 additions & 2 deletions docs/zh_CN/ch1/exchangis_datax_deploy_cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
### 环境准备
DataX引擎是执行Exchangis数据同步任务不可或缺的组件,只有安装部署完成DataX引擎才能够成功执行数据同步任务。同时,确保所部署的机器上有安装DataX。

您在安装部署DataX引擎之前,请按照[Exchangis1.1.2安装部署文档](https://github.com/WeBankFinTech/Exchangis/blob/dev-1.1.2/docs/zh_CN/ch1/exchangis_deploy_cn.md)(TODO 待修改连接地址)安装完成Exchangis1.1.2及相关组件的安装,并确保工程基本功能可用。
您在安装部署DataX引擎之前,请按照[Exchangis安装部署文档](docs/zh_CN/ch1/exchangis_deploy_cn.md)安装完成Exchangis及相关组件的安装,并确保工程基本功能可用。

强烈建议您在执行DataX任务之前,先在该节点使用原生的DataX执行测试任务,以检测该节点环境是否正常。

Expand All @@ -17,7 +17,7 @@ DataX引擎是执行Exchangis数据同步任务不可或缺的组件,只有安

Exchangis1.1.2和Linkis 1.4.0支持的主流DataX版本1.4.6与1.4.7,更高版本可能需要修改部分代码重新编译。

[点击跳转 Release 界面](https://github.com/WeBankFinTech/Exchangis/releases/tag/release-1.1.2)(TODO 待修改连接地址)
[点击跳转 Release 界面](https://github.com/WeBankFinTech/Exchangis/releases/tag/release-1.1.2)

#### 2) 编译打包
如果您想自己开发和编译datax引擎,具体编译步骤如下:
Expand Down
Loading

0 comments on commit a748093

Please sign in to comment.