Dev spark load #40182

gnehil · 2024-08-30T07:50:49Z

Proposed changes

Issue Number: close #xxx

doris-robot · 2024-08-30T07:50:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-08-30T07:57:08Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-30T09:04:04Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-09-18T08:52:03Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh line 162:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |  awk '{print $2}' | tail -1)
       ^----------------^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh line 20:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |  awk '{print $2}' | tail -1)
       ^----------------^ SC2155 (warning): Declare and assign separately to avoid masking return values.

For more information:
  https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- samples/datalake/deltalake_and_kudu/scripts/start_doris.sh.orig
+++ samples/datalake/deltalake_and_kudu/scripts/start_doris.sh
@@ -20,8 +20,8 @@
 export JAVA_HOME=/opt/jdk-17.0.2
 
 cp -r /opt/doris-bin /opt/doris
-echo "trino_connector_plugin_dir=/opt/connectors/" >> /opt/doris/fe/conf/fe.conf
-echo "trino_connector_plugin_dir=/opt/connectors/" >> /opt/doris/be/conf/be.conf
+echo "trino_connector_plugin_dir=/opt/connectors/" >>/opt/doris/fe/conf/fe.conf
+echo "trino_connector_plugin_dir=/opt/connectors/" >>/opt/doris/be/conf/be.conf
 
 /opt/doris/fe/bin/start_fe.sh --daemon
 /opt/doris/be/bin/start_be.sh --daemon
--- samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh
@@ -33,7 +33,6 @@
 hdfs_plugin="ff4a3e3b32dcce27f4df58f17938abde"
 kudu_java_example="1afe0a890785e8d0011ea7342ae5e43d"
 
-
 download_source_file() {
     local FILE_PATH="$1"
     local EXPECTED_MD5="$2"
@@ -79,9 +78,6 @@
 download_source_file "trino-hdfs-435-20240724.tar.gz" "${hdfs_plugin}" "https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724"
 download_source_file "kudu-java-example-1.0-SNAPSHOT.jar" "${kudu_java_example}" "https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724"
 
-
-
-
 if [[ ! -f "jdk-17.0.2/SUCCESS" ]]; then
     echo "Prepare jdk17 environment"
     if [[ -d "jdk-17.0.2" ]]; then
@@ -156,10 +152,9 @@
     touch connectors/trino-delta-lake-435/hdfs/SUCCESS
 fi
 
-
 cd ../
 
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |  awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 | awk '{print $2}' | tail -1)
 
 docker compose -f trinoconnector-compose.yml --env-file trinoconnector-compose.env up -d
 echo "Create hive table ..."
--- samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh
@@ -17,6 +17,6 @@
 # specific language governing permissions and limitations
 # under the License.
 
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |  awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 | awk '{print $2}' | tail -1)
 
 docker compose -f trinoconnector-compose.yml --env-file trinoconnector-compose.env down
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

github-actions · 2024-09-18T09:08:48Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh line 162:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |  awk '{print $2}' | tail -1)
       ^----------------^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh line 20:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |  awk '{print $2}' | tail -1)
       ^----------------^ SC2155 (warning): Declare and assign separately to avoid masking return values.

For more information:
  https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors


'shfmt ' returned error 1 finding the following formatting issues:

----------
--- samples/datalake/deltalake_and_kudu/scripts/start_doris.sh.orig
+++ samples/datalake/deltalake_and_kudu/scripts/start_doris.sh
@@ -20,8 +20,8 @@
 export JAVA_HOME=/opt/jdk-17.0.2
 
 cp -r /opt/doris-bin /opt/doris
-echo "trino_connector_plugin_dir=/opt/connectors/" >> /opt/doris/fe/conf/fe.conf
-echo "trino_connector_plugin_dir=/opt/connectors/" >> /opt/doris/be/conf/be.conf
+echo "trino_connector_plugin_dir=/opt/connectors/" >>/opt/doris/fe/conf/fe.conf
+echo "trino_connector_plugin_dir=/opt/connectors/" >>/opt/doris/be/conf/be.conf
 
 /opt/doris/fe/bin/start_fe.sh --daemon
 /opt/doris/be/bin/start_be.sh --daemon
--- samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh
@@ -33,7 +33,6 @@
 hdfs_plugin="ff4a3e3b32dcce27f4df58f17938abde"
 kudu_java_example="1afe0a890785e8d0011ea7342ae5e43d"
 
-
 download_source_file() {
     local FILE_PATH="$1"
     local EXPECTED_MD5="$2"
@@ -79,9 +78,6 @@
 download_source_file "trino-hdfs-435-20240724.tar.gz" "${hdfs_plugin}" "https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724"
 download_source_file "kudu-java-example-1.0-SNAPSHOT.jar" "${kudu_java_example}" "https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724"
 
-
-
-
 if [[ ! -f "jdk-17.0.2/SUCCESS" ]]; then
     echo "Prepare jdk17 environment"
     if [[ -d "jdk-17.0.2" ]]; then
@@ -156,10 +152,9 @@
     touch connectors/trino-delta-lake-435/hdfs/SUCCESS
 fi
 
-
 cd ../
 
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |  awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 | awk '{print $2}' | tail -1)
 
 docker compose -f trinoconnector-compose.yml --env-file trinoconnector-compose.env up -d
 echo "Create hive table ..."
--- samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh
@@ -17,6 +17,6 @@
 # specific language governing permissions and limitations
 # under the License.
 
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |  awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 | awk '{print $2}' | tail -1)
 
 docker compose -f trinoconnector-compose.yml --env-file trinoconnector-compose.env down
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

…hecking

github-actions · 2024-09-18T09:19:05Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-09-19T09:14:26Z

clang-tidy review says "All clean, LGTM! 👍"

gnehil force-pushed the dev-spark-load branch from 9c6e1a6 to d33628b Compare September 18, 2024 08:51

gnehil requested review from CalvinKirs, morningman and dataroaring as code owners September 18, 2024 08:51

gnehil force-pushed the dev-spark-load branch from d33628b to f35d519 Compare September 18, 2024 09:07

gnehil added 20 commits September 18, 2024 17:12

update TxnCoordinator constructor param

3f9f40a

handle bitmap type convert

cd57e33

rename as bucket load

5975c74

remove test log

fb73764

fix style

6b82b6d

fix rename

cb5f1f9

rename as ingestion load

eeb3d8a

do some refactor

711ac1d

revert format

dfe9adc

add replay state info

13d027e

add replay meta and add schemaVersion to EtlIndex for schema change c…

ae575b6

…hecking

remove useless code

bb1cc28

push task read parquet can go both way of broker and be

2fd20e9

remove useless code

c2e8d41

add regression test

d737039

http action auth

94a7f7f

fix spark load ut

bf13cc0

rename default timeout config

aa427bb

add catalog check and request body example

286bcec

add request example and enrich log information

ab4f715

gnehil added 7 commits September 18, 2024 17:12

change log level to warn

8347844

revert unnecessary changes

14ebf46

redirect to master

affb382

add override annotation

e938c7a

make getLoadTableIds private

51386c5

rearrange fields

a0524ee

fix code mistake

bc7d7e7

gnehil force-pushed the dev-spark-load branch from f35d519 to 0ec0d24 Compare September 18, 2024 09:12

add regression test cases

01b421e

gnehil force-pushed the dev-spark-load branch from 0ec0d24 to 01b421e Compare September 19, 2024 09:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dev spark load #40182

Dev spark load #40182

gnehil commented Aug 30, 2024

doris-robot commented Aug 30, 2024

github-actions bot commented Aug 30, 2024

github-actions bot commented Aug 30, 2024

github-actions bot commented Sep 18, 2024

github-actions bot commented Sep 18, 2024

github-actions bot commented Sep 18, 2024

github-actions bot commented Sep 19, 2024

Dev spark load #40182

Are you sure you want to change the base?

Dev spark load #40182

Conversation

gnehil commented Aug 30, 2024

Proposed changes

doris-robot commented Aug 30, 2024

github-actions bot commented Aug 30, 2024

github-actions bot commented Aug 30, 2024

github-actions bot commented Sep 18, 2024

sh-checker report

github-actions bot commented Sep 18, 2024

sh-checker report

github-actions bot commented Sep 18, 2024

github-actions bot commented Sep 19, 2024

`sh-checker report`

`sh-checker report`